Augmented reality guidance for imaging systems

ABSTRACT

Systems, devices and methods for augmented reality guidance of surgical procedures using multiple head mounted displays or other augmented reality display devices are described. In addition, systems, devices and methods using head mounted displays or other augmented reality display systems for operating surgical robots and/or imaging systems are described.

RELATED APPLICATIONS

This application claims the benefit of and the priority to U.S.Provisional Application Ser. No. 63/158,941, filed Mar. 10, 2021, U.S.Provisional Application Ser. No. 63/163,156, filed Mar. 19, 2021, U.S.Provisional Application Ser. No. 63/173,565, filed Apr. 12, 2021, andU.S. Provisional Application Ser. No. 63/232,376, filed Aug. 12, 2021,the entire contents of each of which is hereby incorporated by referencein their entireties.

TECHNICAL FIELD

Aspects of the present disclosure relate to systems, devices and methodsfor performing a surgical step or surgical procedure with visualguidance using one or more head mounted displays and with display of oneor more imaging studies. Aspects of the present disclosure relate tosystems, devices and methods for operating one or more head mounteddisplays with real-time wireless display of tracking information oftools or instruments registered with the patient's anatomy. Aspects ofthe present disclosure relate to systems, devices and methods foroperating an imaging system with augmented reality display of an imageacquisition area or volume prior to image acquisition.

BACKGROUND

With computer assisted surgery, e.g. surgical navigation or robotics,pre-operative and/or intra-operative imaging studies of the patient canbe used. The imaging studies can be displayed in the operating room onan external computer monitor and the patient's anatomy, e.g. landmarks,can be registered in relationship to the information displayed on themonitor. Since the surgical field is in a different location and has adifferent view coordinate system for the surgeon's eyes than theexternal computer monitor, hand-eye coordination can be challenging forthe surgeon. Image acquisition for pre-operative and/or intra-operativeimaging studies frequently requires the acquisition of imaging dataprior to acquiring the definitive imaging studies, for example used fordiagnostic purposes or image guidance purposes.

SUMMARY

Aspects of the disclosure relate to a system comprising at least onehead mounted display, a robot, wherein the robot comprises an endeffector, a first computing system comprising one or more computerprocessors, and a second computing system comprising one or morecomputer processors, wherein the first computing system is incommunication with the robot, wherein the second computing system is incommunication with the at least one head mounted display, wherein thesecond computing system is configured to display, by the at least onehead mounted display, a virtual user interface comprising at least onevirtual object, wherein the second computing system is configured togenerate a command based at least in part on at least one interactionwith the at least one virtual object displayed in the virtual userinterface, wherein the second computing system is configured to transmitthe command to the first computing system using wireless transmission,wherein the command is configured to cause the first computing system tocontrol the robot for movement, activation, operation, de-activation, orany combination thereof, of a robot component, a robot motor, a robotactuator, a robot drive, a robot controller, a robot hydraulic system, arobot piezoelectric system, a robot switch, the end effector, or anycombination thereof.

In some embodiments, the command is configured to control the endeffector within a predetermined operating boundary, a predeterminedoperating range, a predetermined operating zone, or a predeterminedoperating volume.

In some embodiments, the first computing system is connected to therobot by wire, or wherein the first computing system is connected to therobot by wireless connection.

In some embodiments, the second computing system is connected to the atleast one head mounted display by wire, or wherein the second computingsystem is connected to the at least one head mounted display by wirelessconnection.

In some embodiments, the second computing system is configured todisplay, by the at least one head mounted display, a representation of apredetermined operating boundary, a predetermined operating range, apredetermined operating zone, or a predetermined operating volume of theend effector or an expected outcome following the movement, activation,operation, de-activation or a combination thereof of the robotcomponent, robot motor, robot actuator, robot drive, robot controller,robot hydraulic system, robot piezoelectric system, robot switch, theend effector, or any combination thereof.

In some embodiments, the end effector comprises a physical surgical toolor a physical surgical instrument.

In some embodiments, the first computing system is configured to obtainreal-time tracking information of a component of the robot, the endeffector, a target object, a target anatomic structure of a patient, theat least one head mounted display, a physical tool, a physicalinstrument, a physical implant, a physical object, or any combinationthereof.

In some embodiments, the second computing system is configured to obtainreal-time tracking information of a component of the robot, the endeffector, a target object, a target anatomic structure of a patient, theat least one head mounted display, a physical tool, a physicalinstrument, a physical implant, a physical object, or any combinationthereof.

In some embodiments, the first computing system is configured to obtainreal-time tracking information of a physical tool, a physicalinstrument, or any combination thereof coupled to the robot.

In some embodiments, the second computing system is configured to obtainreal-time tracking information of a physical tool, a physicalinstrument, or any combination thereof coupled to the robot. In someembodiments, the first computing system is configured to wirelesslytransmit the real-time tracking information of the component of therobot, the end effector, a target object, a target anatomic structure ofa patient, the at least one head mounted display, a physical tool, aphysical instrument, a physical implant, a physical object, or anycombination thereof. In some embodiments, the second computing system isconfigured to wirelessly transmit the real-time tracking information ofthe component of the robot, the end effector, a target object, a targetanatomic structure of a patient, the at least one head mounted display,a physical tool, a physical instrument, a physical implant, a physicalobject, or any combination thereof.

In some embodiments, the second computing system is configured fordisplaying, by the at least one head mounted display, a 3D stereoscopicview. In some embodiments, the 3D stereoscopic view is superimposed ontoan anatomic structure of a patient. In some embodiments, the 3Dstereoscopic view comprises a predetermined trajectory of the endeffector, a representation of a predetermined operating boundary of theend effector, a representation of a predetermined operating range of theend effector, a representation of a predetermined operating zone of theend effector, a representation of a predetermined operating volume ofthe end effector, or a combination thereof. In some embodiments, the 3Dstereoscopic view comprises a predetermined trajectory of the endeffector, a representation of a predetermined operating boundary of theend effector, a representation of a predetermined operating range of theend effector, a representation of a predetermined operating zone of theend effector, a representation of a predetermined operating volume ofthe end effector or a combination thereof following the movement,activation, operation, de-activation or combination thereof of the robotcomponent, robot motor, robot actuator, robot drive, robot controller,robot hydraulic system, robot piezoelectric system, robot switch, theend effector or any combination thereof. In some embodiments, the firstcomputing system, the second computing system, or both are configured toturn on or turn off the display of the virtual user interface. In someembodiments, the 3D stereoscopic view comprises a predeterminedtrajectory of the end effector, a representation of a predeterminedoperating boundary of the end effector, a representation of apredetermined operating range of the end effector, a representation of apredetermined operating zone of the end effector, a representation of apredetermined operating volume of the end effector or a combinationthereof prior to executing the command.

In some embodiments, the wireless transmission comprises a Bluetoothsignal, WiFi signal, LiFi signal, a radiofrequency signal, a microwavesignal, an ultrasound signal, an infrared signal, an electromagneticwave or any combination thereof.

In some embodiments, the system comprises two or more head mounteddisplays, wherein the wireless transmission is a multicast, broadcasttransmission or any combination thereof.

In some embodiments, the at least one virtual object comprises one ormore virtual button, virtual field, virtual cursor, virtual pointer,virtual slider, virtual trackball, virtual node, virtual numericdisplay, virtual touchpad, virtual keyboard, or a combination thereof.

In some embodiments, the interaction is a collision detection between aphysical object and the at least one virtual object. In someembodiments, the interaction is a collision detection between a user'sfinger and the at least one virtual object. In some embodiments, theinteraction is a collision detection between a tracked pointer, trackedtool, tracked instrument, or a combination thereof and the at least onevirtual object.

In some embodiments, the interaction with the at least one virtualobject comprises a gaze tracking.

Aspects of the disclosure relate to a system comprising at least onehead mounted display, a robot, wherein the robot comprises an endeffector, a first computing system comprising one or more computerprocessors, wherein the first computing system is in communication withthe robot, a second computing system comprising one or more computerprocessors, wherein the second computing system is in communication withthe at least one head mounted display, wherein the second computingsystem is configured to display, by the at least one head mounteddisplay, a virtual user interface comprising at least one virtualobject, wherein the second computing system is configured to generate anevent message based at least in part on at least one interaction withthe at least one virtual object displayed in the virtual user interface,wherein the second computing system is configured to transmit the eventmessage to the first computing system using wireless transmission,wherein the second computing system is configured to generate a commandbased on the event message, and wherein the command is configured tocause the first computing system to control the robot for movement,activation, operation, de-activation, or any combination thereof, of arobot component, a robot motor, a robot actuator, a robot drive, a robotcontroller, a robot hydraulic system, a robot piezoelectric system, arobot switch, the end effector, or any combination thereof.

In some embodiments, the end effector comprises a scalpel, a saw, acutting tool, a wire, a needle, a pin, a drill, a burr, a mill, areamer, an impactor, a broach, a laser, a radiofrequency device, athermocoagulation device, a cryoablation device, a radioactive probe, aradioactivity emitting device, a pulsed energy emitting device, anultrasonic energy emitting device, a microwave energy emitting device ora combination thereof.

In some embodiments, the command comprises a subcommand, wherein thesubcommand is configured to execute an accept or cancel function of thecommand.

Aspects of the disclosure relate to system, comprising: (a) at least onehead mounted display, at least one camera or scanning device, whereinthe at least one camera or scanning device is configured to trackreal-time information of the at least one head mounted display, of atleast one anatomic structure of a patient, and of at least one physicalsurgical tool or physical surgical instrument, (b) a first computingsystem comprising one or more computer processors, wherein the firstcomputing system is configured to obtain the real-time trackinginformation of the at least one head mounted display, the at least oneanatomic structure of a patient, and the at least one physical surgicaltool or physical surgical instrument, wherein the first computing systemis configured for wireless transmission of the real-time trackinginformation of the at least one head mounted display, the at least oneanatomic structure of the patient, and the at least one physicalsurgical tool or physical surgical instrument, (c) a second computingsystem comprising one or more computer processors, wherein the secondcomputing system is configured for wireless reception of the real-timetracking information of the at least one head mounted display, the atleast one anatomic structure of the patient, and the at least onephysical surgical tool or physical surgical instrument, wherein thesecond computing system is configured to generate a 3D stereoscopicview, wherein the stereoscopic view comprises a 3D representation of theat least one tracked physical surgical tool or physical surgicalinstrument, and wherein the at least one head mounted display isconfigured to display the 3D stereoscopic view.

In some embodiments, the one or more computer processors of the secondcomputing system generate the 3D stereoscopic view for a view angle ofthe head mounted display relative to the at least one anatomic structureof the patient using the real-time tracking information of the at leastone head mounted display.

In some embodiments, the real-time tracking information comprisestracking information of multiple head mounted displays. In someembodiments, the real-time tracking information comprises a head mounteddisplay specific label or tag for each head mounted display, or whereinthe real-time tracking information is labeled for each tracked headmounted display. In some embodiments, the wireless transmission is amulticast or broadcast transmission to the multiple head mounteddisplays.

In some embodiments, the real-time tracking information comprisestracking information of two or more head mounted displays. In someembodiments, the two or more head mounted displays are located indifferent locations. In some embodiments, the real-time trackinginformation comprises a head mounted display label for each head mounteddisplay, wherein each heard mounted display has a different label. Insome embodiments, the real-time tracking information is labeled for eachtracked head mounted display.

In some embodiments, the one or more computer processors of the secondcomputing system generate the 3D stereoscopic view for an interpupillarydistance adjusted for a user wearing the head mounted display.

In some embodiments, the second computing system is communicativelycoupled to the at least one head mounted display.

In some embodiments, the second computing system is integrated with theat least one head mounted display.

In some embodiments, the second computing system is separate from the atleast one head mounted display and is connected to a display unit of theat least one head mounted display using at least one cable.

In some embodiments, the the wireless transmission, the wirelessreception, or both comprise a WiFi signal, a LiFi signal, a Bluetoothsignal or a combination thereof.

In some embodiments, the camera or scanning device is separate from theat least one head mounted display.

In some embodiments, the camera or scanning device is integrated orattached to the at least one head mounted display.

In some embodiments, the wireless transmission comprises sending datapackets comprising the real-time tracking information of the at leastone head mounted display, the at least one anatomic structure of apatient, and the at least one physical surgical tool or physicalsurgical instrument, at a rate of 20 Hz or greater.

In some embodiments, the wireless reception comprises receiving datapackets comprising the real-time tracking information of the at leastone head mounted display, the at least one anatomic structure of apatient, and the at least one physical surgical tool or physicalsurgical instrument, at a rate of 20 Hz or greater.

In some embodiments, the system further comprises a third computingsystem, wherein the third computing system is configured for wirelessreception of the real-time tracking information from the first computingsystem and wherein the third computing system is configured for wirelesstransmission of the real-time tracking information to the secondcomputing system. In some embodiments, the third computing systemcomprises a chain of computing systems configured for wireless receptionand wireless transmission of the real-time tracking information.

In some embodiments, the system further comprises a third computingsystem, wherein the third computing system is communicatively coupled toa second head mounted display, wherein the third computing system isconfigured for wireless reception of the real-time tracking informationof the second head mounted display, the at least one anatomic structureof a patient, and the at least one physical surgical tool or physicalsurgical instrument, wherein the third computing system is configured togenerate a 3D stereoscopic view by the second head mounted display usingthe tracking information of the second head mounted display.

In some embodiments, the tracking information of the second head mountedcomprises a label specific to the second head mounted display foridentifying the tracking information of the second head mounted displayby the third computing system.

In some embodiments, the system further comprises a fourth computingsystem, wherein the fourth computing system is communicatively coupledto a third head mounted display, wherein the fourth computing system isconfigured for wireless reception of the real-time tracking informationof the third head mounted display, the at least one anatomic structureof a patient, and the at least one physical surgical tool or physicalsurgical instrument, wherein the fourth computing system is configuredto generate a 3D stereoscopic view by the third head mounted displayusing the tracking information of the third head mounted display.

In some embodiments, the tracking information of the third head mountedcomprises a label specific to the third head mounted display foridentifying the tracking information of the third head mounted displayby the fourth computing system.

In some embodiments, the system further comprises a fifth computingsystem, wherein the fifth computing system is communicatively coupled toa fourth head mounted display, wherein the fifth computing system isconfigured for wireless reception of the real-time tracking informationof the fourth head mounted display, the at least one anatomic structureof a patient, and the at least one physical surgical tool or physicalsurgical instrument, wherein the fifth computing system is configured togenerate a 3D stereoscopic view by the fourth head mounted display usingthe tracking information of the fourth head mounted display.

In some embodiments, the tracking information of the fourth head mountedcomprises a label specific to the fourth head mounted display foridentifying the tracking information of the fourth head mounted displayby the fifth computing system.

In some embodiments, the real-time tracking information comprises one ormore coordinates. In some embodiments, the one or more coordinatescomprise coordinates of the at least one anatomic structure of thepatient. In some embodiments, the one or more coordinates comprisecoordinates of the at least one physical surgical tool or physicalsurgical instrument. In some embodiments, the one or more coordinatescomprise coordinates of the at least one head mounted display.

In some embodiments, the at least one head mounted display comprises atleast one optical see-through head mounted display.

In some embodiments, the at least one head mounted display comprises atleast one video see-through head mounted display.

In some embodiments, the at least one camera or scanning devicecomprises a laser scanner, a time-of-flight 3D laser scanner, astructured-light 3D scanner, a hand-held laser scanner, a LIDAR scanner,a time-of-flight camera, a depth camera, a video system, a stereoscopiccamera system, a camera array, or a combination thereof.

In some embodiments, the system comprises at least one inertialmeasurement unit. In some embodiments, the at least one inertialmeasurement unit is integrated or attached to the at least one physicalsurgical tool or physical surgical instrument. In some embodiments, theat least one inertial measurement unit is integrated or attached to theat least one anatomic structure of the patient. In some embodiments, theat least one inertial measurement unit is integrated or attached to theat least one head mounted display. In some embodiments, the real-timetracking information of the at least one head mounted display comprisesinformation from the at least one inertial measurement unit.

Aspects of the disclosure relate to a system, comprising (a) two or morehead mounted displays, (b) at least one camera or scanning device,wherein the at least one camera or scanning device is configured totrack real-time information of the two or more head mounted displays, ofat least one anatomic structure of a patient, and of at least onephysical surgical tool or physical surgical instrument, (c) a firstcomputing system comprising one or more computer processors, wherein thefirst computing system is configured to obtain real-time trackinginformation of at least one anatomic structure of a patient, of at leastone physical surgical tool or physical surgical instrument, and of thetwo or more head mounted displays, wherein the tracking information ofthe two or more head mounted displays is labeled for each of the two ormore head mounted displays, wherein the first computing system isconfigured for wireless transmission of the real-time trackinginformation of the at least one anatomic structure of the patient, thetracking information of the at least one physical surgical tool orphysical surgical instrument, and the labeled tracking information ofthe two or more head mounted displays, (d) a second computing system,wherein the second computing system is configured for wireless receptionof the real-time tracking information of the at least one anatomicstructure of the patient, the tracking information of the at least onephysical surgical tool or physical surgical instrument, and the labeledtracking information of the first of the two or more head mounteddisplays, wherein the second computing system is configured to generatea first 3D stereoscopic display specific for a first viewing perspectiveof the first head mounted display using the labeled tracking informationof the first head mounted display, wherein the first head mounteddisplay is configured to display the 3D stereoscopic display, (e) athird computing system, wherein the third computing system is configuredfor wireless reception of the real-time tracking information of the atleast one anatomic structure of the patient, the tracking information ofthe at least one physical surgical tool or physical surgical instrument,and the labeled tracking information of the second of the two or morehead mounted displays, wherein the third computing system is configuredto generate a second 3D stereoscopic display specific for a secondviewing perspective of the second head mounted display using the labeledtracking information of the second head mounted display, wherein thefirst and second stereoscopic displays comprise a 3D representation ofthe at least one physical surgical tool or physical surgical instrument.

In some embodiments, the second computing system is communicativelycoupled to a first of the two or more head mounted displays, and whereinthe third computing system is communicatively coupled to a second of thetwo or more head mounted displays.

Aspects of the disclosure relate to a system of preparing an imagingdata acquisition associated with a patient comprising at least onecomputer processor, an augmented reality display device, an imagingsystem, wherein the at least one computer processor is configured toobtain real-time tracking information of one or more components of theimaging system, wherein the at least one computer processor isconfigured to generate a 3D representation of a surface, a volume orcombination thereof, wherein the 3D representation of the surface,volume or combination thereof is at least in part derived frominformation about a geometry of the one or more components of theimaging system, information about a geometry of the image acquisition,information about one or more image acquisition parameters, or acombination thereof, wherein the at least one computer processor isconfigured to generate an augmented view, the augmented view comprisingthe 3D representation of the surface, volume or combination thereof,wherein the at least one computer processor is configured to display, bythe augmented reality display device, the augmented view at a definedposition and orientation relative to the one or more components of theimaging system, and wherein the position and orientation of theaugmented view is updated based on the real time tracking information ofthe one or more components of the imaging system.

In some embodiments, the 3D representation of the surface, volume orcombination thereof does not contain imaging data from a patient.

In some embodiments, the imaging system is configured to acquire 2D, 3D,or 2D and 3D imaging data of the patient within the 3D representation ofthe surface, volume or combination thereof.

In some embodiments, the at least one computer processor is configuredto generate the 3D representation of the surface, volume or combinationthereof before acquisition of 2D, 3D, or 2D and 3D imaging data of thepatient, or wherein the at least one computer processor is configured todisplay the 3D representation of the surface, volume or combinationthereof before acquisition of 2D, 3D, or 2D and 3D imaging data of thepatient.

In some embodiments, the surface, volume or combination thereofcomprises information about a limit, an edge, a margin, a boundary, acircumference, a perimeter, an envelope or a combination thereof of a2D, 3D, or 2D and 3D imaging data acquisition.

In some embodiments, the at least one computer processor is configuredto generate the surface, volume or combination thereof at least in partfrom information about a geometry of the imaging system, informationabout a geometry of the image acquisition, information about one or moreimage acquisition parameters, or a combination thereof.

In some embodiments, the system is configured to facilitate determininga desired position and orientation of the augmented view, wherein thedesired position and orientation comprises a target anatomic structureof the patient.

In some embodiments, the at least one computer processor is configuredto adjust the augmented view responsive to movement of the one or moretracked components of the imaging system, wherein the adjustment isconfigured to maintain the augmented view at the defined position andorientation relative to the one or more components of the imagingsystem.

In some embodiments, the information about the geometry of the imagingsystem, information about the geometry of the image acquisition,information about one or more image acquisition parameter, or acombination thereof comprises information about one or more imagingsystem components, a geometric relationship between one or more imagingsystem components, a collimator, a grid, an image intensifier, adetector resolution, an x-ray source, an x-ray tube setting, a kVpsetting, an mA setting, an mAs setting, a collimation, a tube—detectordistance, a tube—patient distance, patient—detector distance, apatient—image intensifier distance, a table height relative to a tube, adetector, a table position relative to a tube, a detector, orcombination thereof, a patient position, a C-arm position, orientation,or combination thereof, a gantry position, orientation or combinationthereof, a grid height, a grid width, a grid ratio, a field of view, acenter of a field of view, a periphery of a field of view, a matrix, apixel size, a voxel size, an image size, an image volume, an imagingplane, an image dimension in x, y, z and/or oblique direction, an imagelocation, an image volume location, a scan coverage, a pitch, anin-plane resolution, a slice thickness, an increment, a detectorconfiguration, a detector resolution, a detector density, a tubecurrent, a tube potential, a reconstruction algorithm, a scan range, ascan boundary, a scan limit, a rotational axis of the imaging system, arotational center of the imaging system, a reconstructed slicethickness, a segmentation algorithm, a window, a level, a brightness, acontrast, a display resolution, or a combination thereof.

In some embodiments, the imaging system comprises an x-ray system, afluoroscopy system, a C-arm, a 3D C-arm, a digital tomosynthesis imagingsystem, an angiography system, a bi-planar angiography system, a 3Dangiography system, a CT scanner, an MRI scanner, a PET scanner, a SPECTscanner, a nuclear scintigraphy system, a 2D ultrasound imaging system,a 3D ultrasound imaging system, or a combination thereof.

In some embodiments, the at least one computer processor is configuredto obtain real-time tracking information of the augmented realitydisplay device, an anatomic structure of the patient, a patient tableused with the imaging system, the imaging system, the one or morecomponents of the imaging system, or a combination thereof.

In some embodiments, the system further comprises a camera or scannerconfigured to acquire the real-time tracking information of theaugmented reality display device, the anatomic structure of the patient,the patient table used with the imaging system, the imaging system, theone or more components of the imaging system, or a combination thereof.In some embodiments, the camera or scanner comprises a navigationsystem, a 3D scanner, a LIDAR system, a depth sensor, an IMU or acombination thereof. In some embodiments, the real-time trackinginformation comprises coordinate information of the augmented realitydisplay device, the anatomic structure of the patient, the patient tableused with the imaging system, the imaging system, the one or morecomponents of the imaging system, or a combination thereof. In someembodiments, the real-time tracking information comprises locationinformation of the augmented reality display device, the anatomicstructure of the patient, the patient table used with the imagingsystem, the imaging system, one or more components of the imaging systemcomponents, or a combination thereof. In some embodiments, the camera orscanner comprises a laser scanner, time-of-flight 3D scanner,structured-light 3D scanner, hand-held laser scanner, a time-of-flightcamera or a combination thereof.

In some embodiments, the system is configured to obtain real-timetracking information of the imaging system using intrinsic informationfrom the imaging system, wherein the intrinsic information comprisespose data, sensor data, camera data, 3D scanner data, controller data,drive data, actuator data, end effector data, data from one or morepotentiometers, data from one or more video systems, data from one ormore LIDAR systems, data from one or more depth sensors, data from oneor more inertial measurement units, data from one or moreaccelerometers, data from one or more magnetometers, data from one ormore gyroscopes, data from one or more force sensors, data from one ormore pressure sensors, data from one or more position sensors, data fromone or more orientation sensors, data from one or more motion sensors,position and/or orientation data from step motors, position and/ororientation data from electric motors, position and/or orientation datafrom hydraulic motors, position and/or orientation data from electricand/or mechanical actuators, position and/or orientation data fromdrives, position and/or orientation data from robotic controllers,position and/or orientation data from one or more robotic computerprocessors, or a combination thereof.

In some embodiments, the imaging system is configured to generate anx-ray beam. In some embodiments, the x-ray beam of the imaging system iscone shaped or cylindrical. In some embodiments, the x-ray beam of theimaging system originates from one or more point sources.

In some embodiments, the x-ray beam of the imaging system is collimated.

In some embodiments, the imaging system is configured to generate anx-ray beam, wherein the 3D representation of the surface, volume orcombination thereof comprises information about a limit, an edge, amargin, a boundary, a circumference, a perimeter, an envelope or acombination thereof of the x-ray beam.

In some embodiments, the system further comprising a user interface. Insome embodiments, the user interface comprises a virtual user interface,wherein the virtual interface comprises at least one virtual object. Insome embodiments, the at least one virtual object comprises one or morevirtual button, virtual field, virtual cursor, virtual pointer, virtualslider, virtual trackball, virtual node, virtual numeric display,virtual touchpad, virtual keyboard, or a combination thereof. In someembodiments, the virtual user interface comprises a gesture recognition,gaze recognition, gaze lock, eye tracking, hand tracking, pointertracking, instrument tracking, tool tracking, or a combination thereof.In some embodiments, the at least one computer processor is configuredto generate a command based at least in part on at least one interactionof a user with the at least one virtual object displayed in the virtualuser interface. In some embodiments, the command is configured to move,tilt, or rotate one or more components of the imaging system, one ormore components of a patient table or a combination thereof. In someembodiments, the command is configured to activate, operate, de-activateor a combination thereof a motor, an actuator, a drive, a controller, ahydraulic system, a switch, an electronic circuit, a computer chip, anx-ray tube, an image intensifier, a functional unit of an imagingsystem, or a combination thereof. In some embodiments, the command isconfigured to move or modify a geometry of the imaging system, a patienttable, a geometric relationship between one or more imaging systemcomponents, a collimator, a grid, an image intensifier, a detectorresolution, a setting of the imaging system, a parameter of the imagingsystem, a parameter of the imaging data acquisition, a displayparameter, an x-ray source setting, an x-ray tube setting, a kVpsetting, an mA setting, an mAs setting, a collimation, a tube—detectordistance, a tube—patient distance, patient—detector distance, apatient—image intensifier distance, a table height relative to a tube, adetector, a table position relative to a tube, a detector, a patientposition, a C-arm position, orientation, or combination thereof, agantry position, orientation or combination thereof, a grid height, agrid width, a grid ratio, a field of view, a matrix, a pixel size, avoxel size, an image size, an image volume, an imaging plane, an imagedimension in x, y, z and/or oblique direction, an image location, animage volume location, a scan coverage, a pitch, an in-plane resolution,a slice thickness, an increment, a detector configuration, a detectorresolution, a detector density, a tube current, a tube potential, areconstruction algorithm, a scan range, a scan boundary, a scan limit, areconstructed slice thickness, a segmentation algorithm, a window, alevel, a brightness, a contrast, a display resolution, or a combinationthereof. In some embodiments, the command is configured to set and/ormodify one or more image acquisition parameters of the imaging system.In some embodiments, the command is configured to set, move, and/ormodify a position, orientation, size, area, volume, or combinationthereof of a 2D, 3D or 2D and 3D imaging data acquisition. In someembodiments, the command is configured to set, move, and/or modify oneor more coordinates of the 3D representation. In some embodiments, thecommand is configured to set, move and/or modify a dimension, a size, anarea, a volume or a combination thereof of the 3D representation. Insome embodiments, the setting, moving, and/or modifying of thedimension, size, area, volume or a combination thereof of the 3Drepresentation is configured to set, move and/or modify a 2D, 3D or 2Dand 3D imaging data acquisition to remain at the location of the 3Drepresentation. In some embodiments, the command is configured toactivate, operate, de-activate or a combination thereof a sensor, acamera, a video system, a 3D scanner, a Lidar system, a navigationsystem, a potentiometer, a piezoelectric system, a piezoelectricmechanism, a piezoelectric lock or release system, a controller, adrive, a motor, a hydraulic system, an actuator, or a combinationthereof of the imaging system, an imaging system component, a patienttable or a combination thereof. In some embodiments, the sensorcomprises a depth sensor, inertial measurement unit, accelerometer,magnetometer, gyroscope, force sensor, pressure sensor, position sensor,orientation sensor, motion sensor, or a combination thereof.

In some embodiments, one or more components of the imaging system areattached to or integrated into a robot. In some embodiments, the robotis configured to move one or more components of the imaging system.

In some embodiments, the virtual user interface is configured togenerate an event message triggered by a collision detection. In someembodiments, the system further comprises an event handler configured toprocess the event message. In some embodiments, the event handler isconfigured to generate a command.

In some embodiments, the computing system is configured to generate acommand, wherein the command is triggered by the virtual user interface.

In some embodiments, the system is configured to determine a desiredlocation of the augmented view associated with the imaging system toacquire 2D, 3D, or 2D and 3D imaging data at the desired location.

In some embodiments, the augmented reality display device is a headmounted display, and the augmented view comprises a 3D stereoscopicview.

In some embodiments, the at least one computer processor is configuredto project the 3D stereoscopic view at the coordinates of intended 2D,3D or 2D and 3D imaging data acquisition of the patient. In someembodiments, the location of the 2D, 3D, or 2D and 3D imaging dataacquisition comprises one or more target anatomic structures of thepatient.

Aspects of the disclosure relate to a method of preparing an imageacquisition by an imaging system in a patient comprising: tracking oneor more components of the imaging system in real time; obtaining, by theat least one computer processor, information about a geometry of one ormore components of the imaging system, information about a geometry ofthe image acquisition, information about one or more image acquisitionparameters, or a combination thereof; generating, by the at least onecomputer processor, a 3D representation of a surface, a volume orcombination thereof, wherein the 3D representation of the surface, thevolume or combination thereof is at least in part derived from theinformation about the geometry of the one or more components of theimaging system, information about the geometry of the image acquisition,information about the one or more image acquisition parameters, orcombination thereof; generating, by the at least one computer processor,an augmented view, the augmented view comprising the 3D representationof the surface, volume or combination thereof; and displaying, by anaugmented reality display device, the augmented view, wherein theposition and orientation of the augmented view is defined relative tothe one or more components of the imaging system and is updated based onreal time tracking information of the one or more components of theimaging system.

In some embodiments, the 3D representation of the surface, volume orcombination thereof does not contain imaging data from the patient.

In some embodiments, the imaging system is configured to acquire 2D, 3D,or 2D and 3D imaging data of the patient, and wherein the 2D, 3D, or 2Dand 3D imaging data of the patient are acquired within the 3Drepresentation of the surface, volume or combination thereof.

In some embodiments, the augmented view at the defined position relativeto the one or more components of the imaging system moves in relationwith the tracked one or more components of the imaging system, whereinthe moving facilitates superimposing or aligning the 3D representationwith a target anatomic structure of the patient.

In some embodiments, the position, orientation, position and orientationof the augmented view is adjusted in response to movement of the trackedone or more components of the imaging system.

In some embodiments, the step of generating the augmented view is beforethe step of acquiring 2D, 3D, or 2D and 3D imaging data of the patient,or wherein the step of displaying the augmented view is before the stepof acquiring 2D, 3D, or 2D and 3D imaging data of the patient. In someembodiments, the augmented reality display device is a head mounteddisplay, and wherein the augmented view comprises a 3D stereoscopicview.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative, non-limiting example embodiments will be more clearlyunderstood from the following detailed description taken in conjunctionwith the accompanying drawings.

FIG. 1 shows a non-limiting example of a system configuration fortracking of, for example, anatomic structures, instruments, implants,robots, imaging systems, or any combination thereof and display by oneor more including multiple head mounted displays, e.g. with renderingfor the viewing direction of each head mounted display.

FIG. 2 shows a non-limiting example of a system with modular systemconfiguration, comprising, for example, a tracking module, an instrumentcalibration module, a headset calibration module, an imaging andnavigation module, an augmented reality (AR) wireless networking module,an AR visualization module, and an AR display module.

FIG. 3 shows a non-limiting example of a tracking system, one or morecomputer systems, one or more robots, one or more imaging systems,and/or one or more head mounted displays, wherein, for example, acomputer system can be configured to generate a command, e.g. byinteraction of a user with a virtual object displayed by a virtual userinterface, for activating, de-activating, operating, moving one or moresystems, components, displays etc. of a navigation system, the one ormore head mounted displays, the robot and/or the imaging system.

FIG. 4 shows a non-limiting example of one or more tracking systems, inthis example integrated or attached to one or more head mounteddisplays, one or more computer systems, one or more computer processors,one or more robots, one or more imaging systems, and/or one or more headmounted displays, for example configured for activating, de-activating,operating, moving one or more systems, components, displays etc. of anavigation system, the one or more head mounted displays, the robotand/or the imaging system.

FIG. 5 shows a non-limiting example of one or more tracking systems, inthis example integrated or attached to one or more head mounteddisplays, one or more computer systems, e.g. integrated, connected, orattached to a head mounted display, each with one or more computerprocessors, one or more robots, one or more imaging systems, and/or oneor more head mounted displays, for example configured for activating,de-activating, operating, moving one or more systems, components,displays etc. of a navigation system, the one or more head mounteddisplays, the robot and/or the imaging system.

FIG. 6A shows a non-limiting example of a C-arm system, e.g. a 2D or 3DC-arm, and applications of a virtual display by a head mounted display(HMD) or other augmented reality device.

FIG. 6B shows a non-limiting example of a C-arm system, e.g. a 2D or 3DC-arm, and applications of a virtual display by a head mounted display(HMD) or other augmented reality device, e.g. with display of an outerenvelope or perimeter of an x-ray beam prior to turning on the x-raybeam.

FIG. 6C-6E show non-limiting examples of a 3D C-arm system andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device, e.g. with display of an outer envelopeor perimeter or limit of an intended 3D imaging data acquisition.

FIG. 7A shows a non-limiting example of a radiography (e.g. 2D, 3D),angiography (e.g. 2D, 3D, 4D) or other x-ray based imaging system, andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device.

FIGS. 7B-7C show a non-limiting example of a radiography (e.g. 2D, 3D),angiography (e.g. 2D, 3D, 4D) or other x-ray based imaging system, andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device, e.g. with display of a 3D representationof an outer envelope or perimeter of an x-ray beam prior to turning onthe x-ray beam.

FIGS. 7D-7E show non-limiting examples of a radiography (e.g. 2D, 3D),angiography (e.g. 2D, 3D, 4D) or other x-ray based imaging system andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device, e.g. with display of a 3D representationof an outer envelope or perimeter or limit of a 3D imaging dataacquisition prior to the actual imaging data acquisition.

FIG. 8A shows a non-limiting example of a CT, cone beam CT, spiral CT,MRI system, SPECT system, PET system, or a combination thereof, andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device.

FIG. 8B shows a non-limiting example of a CT, cone beam CT, spiral CT,SPECT system, PET system, or a combination thereof, and applications ofa virtual display by a head mounted display (HMD) or other augmentedreality device, e.g. with display of an outer envelope or perimeter ofan x-ray or energy beam prior to turning on the x-ray or energy beam.

FIG. 8C-8E show non-limiting examples of a CT, cone beam CT, spiral CT,MRI system, SPECT system, PET system, or a combination thereof andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device, e.g. with display of an outer envelopeor perimeter or limit of a 3D imaging data acquisition prior to theactual imaging data acquisition.

FIG. 9 shows a non-limiting example of the use of multiple HMDs or otheraugmented reality display systems for multiple viewer's, e.g. a primarysurgeon, second surgeon, surgical assistant(s) and/or nurses(s)according to some embodiments of the present disclosure.

FIG. 10 shows a non-limiting example of a workflow for segmentation andselect subsequent steps according to some embodiments of the presentdisclosure.

FIG. 11 illustrates a non-limiting example of registering a digitalhologram for an initial surgical step, performing the surgical step andre-registering one or more digital holograms for subsequent surgicalsteps according to some embodiments of the present disclosure.

FIGS. 12A-12C are illustrative, non-limiting examples of virtualobjects, e.g. arbitrary virtual planes, in a hip and a virtual femoralneck cut plane according to some embodiments of the present disclosure,e.g. for use in moving, directing, operating a surgical robot and/or animaging system.

FIG. 13 shows an illustrative, non-limiting example how multiple HMDs orother augmented reality display systems can be used during a surgery,for example by a first surgeon, a second surgeon, a surgical assistantand/or one or more nurses and how a surgical plan can be modified anddisplayed during the procedure by multiple HMDs or other augmentedreality display systems while preserving the correct perspective view ofvirtual data and corresponding live data for each individual operatoraccording to some embodiments of the present disclosure.

FIGS. 14A-14F are illustrative, non-limiting examples of displayingvirtual surgical guides, e.g. a virtual acetabular reaming axis, usingone or more HMDs or other augmented reality display systems and aligninga physical acetabular reamer (e.g. attached to or part of a surgicalrobot) with the virtual reaming axis for placing an acetabular cup witha predetermined cup angle, offset, medial or lateral position and/oranteversion according to some embodiments of the present disclosure.

FIGS. 15A-15D provide illustrative, non-limiting examples of the use ofvirtual surgical guides such as a virtual distal femoral cut blockdisplayed by an HMD and physical surgical guides such as physical distalfemoral cut blocks (e.g. attached to or part of a surgical robot) forknee replacement according to some embodiments of the presentdisclosure.

FIGS. 16A-16C provide an illustrative, non-limiting example of the useof virtual surgical guides such as an AP femoral cut block displayed byan HMD and physical surgical guides such as physical AP cut blocks (e.g.attached to or part of a surgical robot) for knee replacement accordingto some embodiments of the present disclosure.

FIGS. 17A-17F provide a illustrative, non-limiting examples of the useof virtual surgical guides such as a virtual proximal tibial cut guidedisplayed by an HMD and physical surgical guides such as physicalproximal tibial cut guide (e.g. attached to or part of a surgical robot)according to some embodiments of the present disclosure.

FIG. 18 shows a wooden board with 25 squares and four 4.0×4.0 cm opticalmarkers.

FIG. 19 shows an illustrative, non-limiting example of registration offour cubes in relationship to four optical markers using the imagecapture system of an HMD.

FIG. 20 shows an illustrative, non-limiting example of optical markers.

FIG. 21 shows an illustrative, non-limiting example of detection ofoptical markers using the image capture system of an HMD.

FIG. 22 shows an illustrative, non-limiting example of the accuracy ofdetecting an optical marker using a video camera integrated into an HMD.

FIGS. 23A-23E show an illustrative, non-limiting example for placing anintended path of a pedicle screw using a virtual interface. Theplacement can be executed via free hand technique or, for example, usinga surgical robot, e.g. with a robotic tool or instrument sleeve or drillguide.

FIGS. 24A-24B provide illustrative, non-limiting examples of one or moreaugmented reality HMD displays including a virtual user interface forvirtual placing, sizing, fitting, selecting and aligning of virtualpedicle screws and including OHMD displays for guidance of spinalinstruments and implants.

FIGS. 25A-25B provide illustrative, non-limiting examples of one or moreaugmented reality OHMD displays for virtual placing, sizing, fitting,selecting and aligning of implant components for free hand or roboticsurgery.

DETAILED DESCRIPTION

The following description provides exemplary embodiments only, and isnot intended to limit the scope, applicability, or configuration of thedisclosure. Rather, the following description of the exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing one or more exemplary embodiments. It willbe understood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope ofthe presently disclosed embodiments.

Aspects of the present disclosure provide, among other things, systems,devices and methods for a simultaneous visualization of live data of thepatient and digital representations of virtual data such as virtualoperating ranges, virtual operating areas, virtual operating volumes,e.g. for robots and/or imaging system, virtual image acquisition ranges,virtual image acquisition areas, virtual image acquisition volumes, e.g.for imaging systems, virtual cuts and/or virtual surgical guidesincluding cut blocks, drilling guides, one or more virtual axes, one ormore virtual planes or a combination thereof through a head mounteddisplay (HMD) or other augmented reality display system. In someembodiments, the system can include one or more HMDs or other augmentedreality display systems, one or more processors and one or more userinterfaces. In some embodiments, the surgical site including live dataof the patient, the HMD, and the virtual data are registered in a commoncoordinate system. In some embodiments, the virtual data aresuperimposed onto and aligned with the live data of the patient. In someembodiments, the head mounted display is a see-through HMD, e.g. a videosee-through HMD or an optical see-through HMD. Unlike virtual realityhead systems that blend out live data, the HMD can allow the surgeon tosee the live data of the patient through the HMD, e.g. the surgicalfield, while at the same time observing virtual data of the patientand/or virtual surgical instruments or implants with a predeterminedposition and/or orientation using the display of the HMD unit.

In any of the embodiments of the disclosure, a video see-through HMD oran optical see-through HMD can be used. In any of the embodiments of thedisclosure, other augmented reality display devices can be used, e.g. inconjunction with an HMD or instead of an HMD, for example a tablet, e.g.an iPad (Apple, Cupertino, Calif.) or Surface (Microsoft, Redmond,Wis.), or a smart phone, e.g. an iPhone (Apple Cupertino, Calif.). Whenother augmented reality display devices are used, they can comprise anoptional video camera or scanner, e.g. 3D scanner, including, forexample, a LIDAR system, for scanning physical objects, such as animaging system, a surgical robot (also referred herein as robotic systemor surgical robotic system), an OR table, a patient on the OR table, animaging system table, a patient on the imaging system table, a physicaltool, a physical instrument, an end effector etc. The augmented realitydisplay can comprise a composite or mixed reality or augmented realitydisplay of the video feed and virtual devices or virtual objects, e.g. avirtual end effector or a 3D representation of an x-ray beam or intendedimage acquisition. Any virtual devices, virtual surgical guide, virtualtool or instrument, or virtual object known in the art or described inthe specification can be co-displayed with the video feed or videoimages. The virtual devices, virtual surgical guide, virtual tool orinstrument, or virtual object known in the art or described in thespecification can be displayed in conjunction with the video feed andcan optionally be registered with the physical objects, devices (e.g. animaging system or a surgical robot) or a physical patient or targetanatomic structure included in the video feed or video images. Any ofthe registration techniques described in the specification or known inthe art can be used. The terms mixed reality and augmented reality asused throughout the disclosure can be used interchangeably. In any ofthe illustrations, the term HMD (i.e. head mounted display) can be usedinterchangeably with an augmented reality display device, mixed realitydisplay device, e.g. a tablet or smart phone. In some embodiments, theterms head mounted display, HMD, augmented reality display device, mixedreality display device can be used interchangeably.

In some embodiments, an operator such as a surgeon can look through anHMD observing physical data or information on a patient, e.g. a surgicalsite or changes induced on a surgical site, while pre-existing data ofthe patient are superimposed onto the physical visual representation ofthe live patient. Systems, methods and techniques to improve theaccuracy of the display of the virtual data superimposed onto the livedata of the patient are described in International Patent ApplicationNo. PCT/US2018/012459, which is incorporated herein by reference in itsentirety.

Methods and systems of registration and cross-referencing includingregistration and cross-referencing surgical sites and one or more HMDsor other augmented reality display systems (e.g. using inside-outtracking techniques, outside-in tracking techniques, and combinationsthereof) such as the ones described in PCT International ApplicationSerial Nos. PCT/US2017/021859, PCT/US2018/013774 and PCT/US2019/015522can be used. Methods, techniques, and systems of displaying virtual datain various surgical, medical or dental applications using one or moreHMDs or other augmented reality display systems such as the onesdescribed in PCT International Application Serial Nos.PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/61698,PCT/US2019/015522, and U.S. Pat. No. 9,861,446 can be used. Theseapplications are hereby incorporated by reference in their entireties.

Aspects of the present disclosure relate to systems, devices and methodsfor performing a surgical step or surgical procedure with visualguidance using a head mounted display. In some embodiments, the headmounted display can be a see-through head mounted display, e.g. anoptical see-through head mounted display, for example for augmentedreality applications. In some embodiments, the head mounted display canbe a non-see through head mounted display, e.g. video-see through type,for virtual reality applications, optionally with video displayincluding video streaming of live data from the patient, e.g. video feedfrom a camera integrated into, attached to, or separate from the headmounted display. The head mounted display can provide surgical guidancein a mixed reality environment.

Some aspects of the disclosure relate to a system for performing asurgical procedure, the system comprising: a processor; a see-throughhead mounted display or other augmented reality display device; and amarker attached to a patient, wherein, the system is configured togenerate a 3D stereoscopic view or augmented view of a virtual surgicalguide, wherein the virtual surgical guide is a placement indicator atone or more predetermined coordinates indicating a predeterminedposition, predetermined orientation or combination thereof for aligninga physical surgical tool or a physical surgical instrument, wherein thesystem is configured to display the 3D stereoscopic view by the seethrough head mounted display onto the patient, e.g. a patient's spine, apatient's joint, a patient's tooth, gum, dental structure or combinationthereof. The processor can be configured to determine a distance betweenone or more predetermined coordinates of the virtual surgical guide andthe see through head mounted display, wherein the one or morepredetermined coordinates of the virtual surgical guide can bereferenced to or based on the marker. In some embodiments, the processorcan be configured to adjust at least one focal plane, focal point,convergence or combination thereof of the display of the 3D stereoscopicview based on a determined distance, e.g. using inside-out or outside-intracking or a combination thereof. In some embodiments, the system canbe configured to track, e.g. in real-time, a robot component, an endeffector, an imaging system component, or a combination thereof.Inside-out tracking can comprise tracking, for example, a head mounteddisplay, an augmented reality display device, an anatomic structure ofthe patient, a patient table used with the imaging system, an imagingsystem, one or more components of the imaging system, a surgicalinstrument, a surgical tool, an implant, a surgical robot, a robotintegrated with or part of the imaging system, a physical object or anycombination thereof using at least one camera, scanner (includingnavigation systems, LIDAR systems etc.) or combination thereofintegrated into a head mounted display or augmented reality displaydevice. Outside-in tracking can comprise tracking, for example, a headmounted display, an augmented reality display device, an anatomicstructure of the patient, a patient table used with the imaging system,an imaging system, one or more components of the imaging system, asurgical instrument, a surgical tool, an implant, a surgical robot, arobot integrated with or part of the imaging system, a physical objector any combination thereof using at least one camera, scanner (includingnavigation systems, LIDAR systems etc.) or combination thereof separatefrom a head mounted display or augmented reality display device. In someembodiments, the system comprises one or more markers. In someembodiments, the marker can be configured to reflect or emit light witha wavelength between 380 nm and 700 nm. In some embodiments, the markercan be configured to reflect or emit light with a wavelength greaterthan 700 nm. In some embodiments, the marker can be a radiofrequencymarker, or the marker can be an optical marker, wherein the opticalmarker can include a geometric pattern. In some embodiments, the one ormore markers can comprise at least one marker attached to the patient,at least one marker attached to a see through head mounted display, atleast one marker attached to a structure in the operating room or anycombination thereof.

In some embodiments, the system can be configured to determine one ormore coordinates using one or more cameras.

In some embodiments, the one or more cameras detect light with awavelength between 380 nm and 700 nm. In some embodiments, the one ormore cameras detect light with a wavelength above 700 nm. In someembodiments, the one or more cameras detect light with a wavelengthbetween 380 nm and 700 nm, above 700 nm or a combination thereof.

In some embodiments, the system comprises at least one camera, scanner,3D scanner, LIDAR system, depth sensor, inertial measurement unit (IMU),oscilloscope, gyroscope, or a combination thereof integrated into orattached to the head mounted display. In some embodiments, at least onecamera, scanner, 3D scanner, LIDAR system, depth sensor, IMU,oscilloscope, gyroscope or a combination thereof is separate from thehead mounted display. In some embodiments, the one or more camera,scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope,gyroscope or a combination thereof are configured to determine theposition, orientation, or position and orientation of the marker. Insome embodiments, the one or more camera, scanner, 3D scanner, LIDARsystem, depth sensor, IMU, oscilloscope, gyroscope or a combinationthereof are configured to determine one or more coordinates of themarker. In some embodiments, the one or more camera, scanner, 3Dscanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or acombination thereof are configured to track the one or more coordinatesof the marker during movement of the marker. In some embodiments, theone or more camera, scanner, 3D scanner, LIDAR system, depth sensor,IMU, oscilloscope, gyroscope or a combination thereof are configured todetermine one or more coordinates of the patient directly (e.g.markerless), e.g. by detecting select anatomic landmarks and/orstructures and/or surfaces, e.g. a spinal structure and/or surface,articular structure and/or surface, tooth and/or surface, gum and/orsurface, dental structure and/or surface, other structure and/or surfaceor body tissues. In some embodiments, the one or more camera, scanner,3D scanner, LIDAR system, depth sensor, IMU or a combination thereof areconfigured to determine one or more coordinates of the see through headmounted display.

In some embodiments, the system is configured to track the one or morecoordinates of the see through head mounted display during movement ofthe patient, the see through head mounted display, or the patient andthe see through head mounted display. The movement of the patient canbe, for example, the movement of a spine, one or more spinal elements, ahead, a joint, one or more articular surfaces, a mandible, a maxilla, atooth.

In some embodiments, the system comprises one or more processors. Insome embodiments, the one or more processors are configured to generatethe 3D stereoscopic view of the virtual surgical guide, virtual display,e.g. virtual axis, virtual plane, virtual operating range, area orvolume (e.g. of a robot and/or imaging system), virtual imageacquisition range, area or volume (e.g. of an imaging system). In someembodiments, the one or more processors are configured to determine thedistance between the one or more predetermined coordinates of thevirtual surgical guide, virtual display, e.g. virtual axis, virtualplane, virtual operating range, area or volume (e.g. of a robot and/orimaging system), virtual image acquisition range, area or volume (e.g.of an imaging system) and the see through head mounted display. In someembodiments, the one or more processors are configured to track one ormore coordinates of at least one or more markers, one or more anatomicstructures, one or more see through head mounted displays, orcombinations thereof during movement of the patient, the see throughhead mounted display or the patient and the see through head mounteddisplay. In some embodiments, the one or more processors are configuredto determine the distance between the one or more predeterminedcoordinates of the virtual surgical guide, virtual display, e.g. virtualaxis, virtual plane, virtual operating boundary, virtual operatingrange, area or volume (e.g. of a robot and/or imaging system), virtualimage acquisition range, area or volume (e.g. of an imaging system) andthe see through head mounted display during movement of the markerand/or the anatomic structure, movement of the see through head mounteddisplay, or movement of the marker and/or the anatomic structure and thesee through head mounted display. In some embodiments, one or moreprocessors are configured to adjust at least one focal plane, focalpoint, convergence or combination thereof based on a change in adetermined distance, e.g. from an HMD to a surgical site and/or anatomicstructure. In some embodiments, one or more computer processors and/orcomputing systems, e.g. a first, second, third, fourth, etc. computerprocessor and/or computing systems are configured to display, e.g. by acomputer monitor and/or one or more head mounted displays, a virtualsurgical guide, e.g. a virtual axis, virtual plane, a virtual operatingrange, virtual operating area or virtual operating volume (e.g. of arobot and/or imaging system), a virtual image acquisition range, virtualimage acquisition area or virtual image acquisition volume (e.g. of animaging system). A first computer processor and/or computing system canbe configured to communicate (e.g. via direct cable connection orwireless connection) to a robot and/or an imaging system or to becommunicatively coupled to the robot and/or imaging system. A secondcomputer processor and/or computing system can be configured tocommunicate (e.g. via direct cable connection or wireless connection) toone or more head mounted displays or to be communicatively coupled tothe head mounted display(s). In some embodiments, the physical surgicaltool or physical surgical instrument can be configured to effect atissue removal in the patient. A tissue removal can be, for example, anosteotomy of a bone (e.g. using an osteotome, as used in spinaldeformity operations or in articular procedures), a pinning, drilling,milling, reaming, broaching, impacting and/or cutting of a bone using,for example, a pin, drill, mill, reamer, broach, impactor, and/or saw orsawblade, optionally attached to or integrated into a robot (e.g.hand-held or attached to an OR table) or a robotic arm. In someembodiments, a robotic end effector can be configured to effect a tissueremoval or tissue alteration in the patient. The tissue removal ortissue alteration can be a removal of bone or a removal of cartilage ora removal of bone and cartilage, or a removal of a tooth and/or dentaltissue, a tissue ablation, a tissue coagulation, a cell transfer, animplantation etc. Examples include a thermocoagulation, a cryoablation,a cutting with a scalpel or other cutting device. A tissue removal oralteration can be a removal or addition/supplementation of bone, bonetissue, cartilage, dental tissue, gum, gum tissue, brain, brain tissue,organ (e.g. liver, spleen, kidneys, bowel, stomach, heart, lung,thyroid, parathyroid tissue), skin, dermal tissue, subcutaneous tissue,or any combination thereof.

Aspects of the present disclosure relate to devices and methods forperforming a surgical step or surgical procedure with visual guidanceusing one or more head mounted displays and with display of one or moreimaging studies, e.g. x-rays, Panorex views, CT scan (for example,spiral CT, cone beam CT), MRI scan, ultrasound scan, PET scan, SPECTscan or a combination thereof.

Bluetooth

In some embodiments, the device can comprise a Bluetooth transmitterand/or receiver.

Bluetooth can be a packet-based protocol with a master/slavearchitecture. One master can communicate with multiple slaves in apiconet. A master Bluetooth device can communicate with multiple devicesin a piconet. The devices can switch roles, by agreement, and the slavecan become the master (for example, a headset initiating a connection toa phone can begin as master—as an initiator of the connection—but maysubsequently operate as the slave).

Bluetooth can be a layer protocol architecture comprising coreprotocols, cable replacement protocols, telephony control protocols, andadopted protocols. The device can, in some embodiments, employhigh-speed Bluetooth protocols. The device can comprise an interfacebetween a server and the device using a Bluetooth device. The interfacecan be HCI (Host Controller Interface).

The Host Controller Interface can provide a command interface for thecontroller and for the link manager, which can allow access to thehardware status and control certain registers. This interface canprovide an access layer for all Bluetooth devices. The HCI layer of themachine can exchange commands and data with the HCI firmware present inthe Bluetooth device. The HCI can, in some embodiments, automaticallydiscover other Bluetooth devices that are within the coverage radius.

The hardware that constitutes a Bluetooth device, including theBluetooth device that can optionally be within the device, can includetwo parts: a radio device, responsible for modulating and transmittingthe signal and a digital controller. These specific parts can, in someembodiments be physically separate and can in other embodiments bephysically together.

The digital controller can, in some embodiments, be a computer processoror a central processing unit (CPU). In some embodiments, the computerprocessor CPU can run a Link Controller; and interfaces with the hostdevice, such as the Host Controller Interface. The Link Controller canbe responsible for the processing of the baseband and the management ofARQ and physical layer FEC protocols. The computer processor or the CPUcan, in some embodiments, handle the transfer functions (bothasynchronous and synchronous), audio coding, and data encryption. Thecomputer processor or CPU of the device can, in some embodiments, beresponsible for performing the instructions related to the Bluetooth ofthe host device, in order to simplify its operation. For the performanceof specific instructions related to the Bluetooth of the host device,the computer processor or the CPU can run software called Link Managerthat has the function of communicating with other devices through theLMP protocol.

The Link Manager can, in some embodiments, establish the connectionbetween devices. For example, the Link Manager can establish theconnection between the devices. The Link Manager can be responsible forthe establishment, authentication and configuration of the link. TheLink Manager can furthermore find other managers and communicates withthem due to the management protocol of the LMP link.

The Link Manager Protocol can comprise a number of PDUs (Protocol DataUnits) that can be sent from one device to another. The following is alist of supported services:

-   -   1) Transmission and reception of data    -   2) Name request    -   3) Request of the link addresses    -   4) Establishment of the connection    -   5) Authentication    -   6) Negotiation of link mode and connection establishment

The system, when in discoverable mode, can transmit the followinginformation on demand:

-   -   1) Device name    -   2) Device class    -   3) List of services    -   4) Technical information (for example: device features,        manufacturer, Bluetooth specification used, clock offset)

The system can have a unique 48-bit address. The system can have afriendly Bluetooth name, which can be set by the user. This name canappear when another user scans for devices and in lists of paireddevices.

During pairing between the server and the system the two can establish arelationship by creating a shared secret or a link key. If both devicesstore the same link key, they can be paired or bonded. The following arepairing mechanisms that can be used in some embodiments of thedisclosure:

-   -   1) Legacy pairing, wherein each device must enter a PIN code;        pairing is only successful if both devices enter the same PIN        code. Legacy has the following authentication mechanisms:        -   a. Limited input devices, wherein the devices have a fixed            PIN, for example “1111” or “2222”, that are hard-coded into            the device        -   b. Numeric input devices, wherein the user can enter a            numeric value up to 16 digits in length        -   c. Alpha-numeric input devices wherein the user can enter            full UTF-8 text as a PIN code    -   2) Secure Simple Pairing (SSP), using a public key cryptography,        and certain modifications can help protect against man in the        middle, or MITM attacks. SSP has the following authentication        mechanisms:        -   a. Just works: This method functions with no user            interaction. However, the device may prompt the user to            confirm the pairing process.        -   b. Numeric comparison: The devices being paired display a            6-digit numeric code. The user can compare the numbers to            ensure they are the exact same. If the comparison succeeds,            the user(s) can confirm pairing on the device(s) that can            accept an input. This method provides MITM protection,            assuming the user confirms on both devices and actually            performs the comparison properly.        -   c. Passkey Entry: This mechanism can be used between a            device with a display and a device with numeric keypad entry            (such as a keyboard), or two devices with numeric keypad            entry. In the first case, the display presents a 6-digit            numeric code to the user, who then enters the code on the            keypad. In the second case, the user of each device enters            the same 6-digit number.        -   d. Out of band (OOB): This method uses an external means of            communication, such as near-field communication (NFC) to            exchange information used in the pairing process. Pairing is            completed using the Bluetooth radio, but requires            information from the OOB mechanism.

In some embodiments, the device comprises a Bluetooth transmitter and/orreceiver wherein the Bluetooth transmitter and/or receiver is configuredto work in conjunction with an augmented reality surgical guidancesystem, a surgical navigation system, a robot, a robotic system, and/ora handheld robot.

In some embodiments, the Bluetooth transmitter and/or receiver and theestablished connection between the Bluetooth transmitter and/or receiverand the augmented reality surgical guidance system, surgical navigationsystem, robot, robotic system, and/or handheld robot can work inconjunction with one or more on/off switches and/or one or morepotentiometers, e.g. digital potentiometers, and/or one or morerheostats and/or one or more actuators to regulate the speed of themovement of the saw blade or movement of the drill bit or to providehaptic feedback.

For example, in cases where the augmented reality surgical guidancesystem, surgical navigation system, robot, robotic system, and/orhandheld robot detects a movement of a surgical instrument or tooldeviating from an intended surgical axis, target, target area, targetvolume, tissue resection target, area, volume (e.g. bone or tissueremoval or resection, e.g. with a bone drill or bone saw) by a specificdistance in any direction in one or more dimensions, the augmentedreality surgical guidance system, surgical navigation system, robot,robotic system, and/or handheld robot can transmit information to theBluetooth receiver which can regulate the Bluetooth switch, includingboth a transmitter and receiver, to activate an on/off switch and/or apotentiometer, e.g. digital, and/or a rheostat and/or a specificactuator for haptic feedback. In cases where the augmented realitysurgical guidance system, surgical navigation system, robot, roboticsystem, and/or handheld robot detects a movement of a drill or saw orother power tool that approaches, for example, a specific anatomicalstructure or safe zone, the augmented reality surgical guidance system,surgical navigation system, robot, robotic system, and/or handheld robotcan similarly work in conjunction with the Bluetooth switch within thedevice attached to the drill or saw to adjust, control, and/or regulatean on/off switch and/or a potentiometer and/or a rheostat and/or aspecific actuator for haptic feedback. The same concept can similarlywork for turning on or increasing the speed of the movement of the sawblade or the drill bit or other power tool or instrument whenapproaching certain anatomic structures.

The Bluetooth switch, Bluetooth receiver, and/or Bluetooth transmittercan, in some embodiments, employ low latency Bluetooth in order toprovide instant saw or drill speed regulation or instant hapticfeedback.

WiFi

In some embodiments, the device comprises a WiFi transmitter and/orreceiver. In some embodiments, the device can comprise WiFi capability.Different versions of WiFi can be used including but not limited to:802.11a, 802.11b, 802.11g, 802.11n (Wi-Fi 4[40]), 802.11h, 802.11i,802.11-2007, 802.11-2012, 802.11ac (Wi-Fi 5[40]), 802.11ad, 802.11af,802.11-2016, 802.11ah, 802.11ai, 802.11aj, 802.11aq, 802.11ax (Wi-Fi6[40]), and 802.11ay. In some embodiments, the device comprises a WiFitransmitter and/or receiver wherein the WiFi transmitter and/or receiveris configured to work in conjunction with a surgical guidance system. Insome embodiments, the system can include routers that can be configuredfor intranet and internet connections.

In some embodiments, the system can utilize several distinct radiofrequency ranges. For example, the system utilizes the 802.11 standard,it can include distinct radio frequencies ranges for use in Wi-FIcommunications such as: 900 MHz, 2.4 GHz, 5 GHz, 5.9 GHz, and 60 GHzbands. Each frequency or range can have a multitude of channels.

In some embodiments, the system and/or device's Wi-Fi can be part of theIEEE 802 protocol family. In some embodiments, the system and/or devicecan comprise one or more transmitters. WiFi transmitters are low powerdevices.

In some embodiments, the system and/or device can comprise one or moreantennas. The system and/or device can comprise an access pointcompliant with 802.11b and/or 802.11g. Using the stock omnidirectionalantenna can have a range of 100 m (0.062 mi). The identical radio withan external semi parabolic antenna (15 dB gain) with a similarlyequipped receiver at the far end can have a range over 20 miles.

In some embodiments, the system and/or device can comprisemultiple-input and multiple-output. The system and/or device includingbut not limited to standards such as IEEE 802.11n and IEEE 802.11ac, cancomprise multiple antennas for extended range and higher speeds.

In some embodiments, the WiFi can comprise Local Area Networks (LAN). Insome embodiments, the device can include one or more access points. Awireless access point can connect a group of wireless devices to anadjacent wired LAN. In some embodiments, the device can include one ormore wireless adapters. Wireless adapters can allow devices to connectto a wireless network

In some embodiments, the device can include one or more routers.Wireless routers can integrate a Wireless Access Point, Ethernet switch,and internal router firmware application that provides IP routing, NAT,and DNS forwarding through an integrated WAN-interface.

In some embodiments, the device can include one or more wireless networkbridges. Wireless network bridges can act to connect two networks toform a single network at the data-link layer over Wi-Fi. The mainstandard is the wireless distribution system (WDS).

Wireless bridging can connect a wired network to a wireless network.

In some embodiments, the device can include one or more securityfeatures. Security features can be any security standard known in theart.

In some embodiments, the WiFi transmitter and/or receiver and theestablished connection between the WiFi transmitter and/or receiver andthe augmented reality surgical guidance system can work in conjunctionwith one or more on/off switches and/or one or more potentiometersand/or one or more rheostats and/or one or more actuators to regulatethe oscillation of a saw blade or movement of a drill bit or to providehaptic feedback.

For example, in cases where the augmented reality surgical guidancesystem detects a movement of a drill or saw or other power tool orinstrument deviating from the intended cut/drill surgical axis, thesurgical guidance system can regulate a WiFi switch, including both atransmitter and receiver, to activate an on/off switch and/or apotentiometer, e.g. digital, and/or a rheostat and/or a specificactuator for haptic feedback. In cases where the surgical guidancesystem detects a movement of a drill or saw or other power tool orinstrument that approaches, for example, a specific anatomical structureor safe zone, the surgical guidance system can similarly work inconjunction with a WiFi switch within the device attached to a drill orsaw or other power tool or instrument to activate an on/off switchand/or a potentiometer and/or a rheostat and/or a specific actuator forhaptic feedback. The same concept can similarly work for turning on orincreasing the speed of the movement of a saw blade or a drill bit orother power tool or instrument when approaching certain anatomicstructures.

LiFi

In some embodiments, the device can comprise a LiFi transmitter and/orreceiver. In some embodiments, the device can comprise LiFi capability.LiFi can use light from light-emitting diodes (LEDs) as a medium todeliver networked, mobile, high-speed communication.

In some embodiments, the system can comprise visible lightcommunications (VLC). VLC works by switching the current to the LEDs offand on at very high speeds.

In some embodiments, the system can comprise Bg-Fi. Bg-Fi can be a Li-Fisystem consisting of an application for a mobile device, and a simpleconsumer product device, with color sensor, microcontroller, andembedded software. Light from the mobile device display communicates tothe color sensor on the consumer product, which converts the light intodigital information. Light emitting diodes enable the consumer productto communicate synchronously with the mobile device.

In some embodiments, the Li-Fi system can be wireless and can use 802.11protocols. In some embodiments, the LiFi system can use ultraviolet,infrared and visible light communication. One part of the visible lightcommunication can be designed from communication protocols establishedby the IEEE 802 workgroup. The IEEE 802.15.7 standard can, in someembodiments, define the physical layer (PHY) and media access control(MAC) layer. The modulation formats recognized for PHY I and PHY II areon-off keying (OOK) and variable pulse position modulation (VPPM). TheManchester coding used for the PHY I and PHY II layers can include theclock inside the transmitted data by representing a logic 0 with an OOKsymbol “01” and a logic 1 with an OOK symbol “10”, all with a DCcomponent. The DC component avoids light extinction in case of anextended run of logic 0's.

The use of LiFi provides additional benefits as the light waves areunlikely to affect or hinder the efficiency of a medical procedure ormedical devices.

In some embodiments, the device can comprise a LiFi transmitter and/orreceiver wherein the LiFi transmitter and/or receiver is configured towork in conjunction with a surgical guidance system. In someembodiments, the LiFi transmitter and/or receiver and the establishedconnection between the LiFi transmitter and/or receiver and theaugmented reality surgical guidance system can work in conjunction withone or more on/off switches and/or one or more potentiometers and/or oneor more rheostats and/or one or more actuators to regulate theoscillation of a saw blade or movement of a drill bit or to providehaptic feedback.

For example, in cases where an augmented reality surgical guidancesystem detects a movement of a drill or saw deviating from the intendedcut/drill surgical axis, the surgical guidance system can regulate theLiFi switch, including both a transmitter and receiver, to activate anon/off switch and/or a potentiometer and/or a rheostat and/or a specificactuator for haptic feedback. In cases where the surgical guidancesystem detects a movement of a drill or saw that approaches, forexample, a specific anatomical structure or safe zone, the surgicalguidance system can similarly work in conjunction with the LiFi switchwithin a device attached to or integrated into a drill or saw toactivate an on/off switch and/or a potentiometer and/or a rheostatand/or a specific actuator for haptic feedback. The same concept cansimilarly work for turning on or increasing the speed of the movement ofa saw blade or a drill bit when approaching certain anatomic structures.

In some embodiments, other forms of wireless data transmission known inthe art can be used, not only Bluetooth, Wifi, Lifi, but also, but notlimited to, a radiofrequency signal, a microwave signal, an ultrasoundsignal, an infrared signal, an electromagnetic wave or a combinationthereof. Any form of wireless data transmission known in the art can beused in any of the embodiments.

In some embodiments, the system comprises at least one camera, videosystem and/or scanner (e.g. a 3D scanner, a laser scanner, a LIDARsystem or LIDAR scanner), a depth sensor, an IMU or a combinationthereof integrated into or attached to the see through head mounteddisplay. In some embodiments, at least one camera, video system and/orscanner (e.g. a 3D scanner, a laser scanner, a LIDAR system or LIDARscanner), a depth sensor, an IMU or a combination thereof is/areseparate from the head mounted display. In some embodiments, one or morecamera, video system, scanner, 3D scanner, LIDAR system, depth sensor,IMU or a combination thereof are configured to determine the position,orientation, or position and orientation of a marker, a surface, and/ora tissue. In some embodiments, one or more camera, video system,scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combinationthereof are configured to determine one or more coordinates of a marker,a surface, and/or a tissue. In some embodiments, one or more camera,video system, scanner, 3D scanner, LIDAR system, depth sensor, IMU,oscilloscope, gyroscope or a combination thereof are configured to trackone or more coordinates of a marker, a surface, and/or a tissue duringmovement of the marker, the surface, and/or the tissue. In someembodiments, one or more camera, video system, scanner, 3D scanner,LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or acombination thereof are configured to determine one or more coordinatesof a see through head mounted display. In some embodiments, one or moremarkers can be attached to or integrated into a physical instrument, aphysical tool, a physical trial implant, a physical implant, a physicaldevice, one or more HMDs or other augmented reality display systems, arobot, a robotic arm, a handheld robot, an end effector, an imagingsystem, one or more components of an imaging system or a combinationthereof.

Imaging Systems

The term imaging system as used throughout the specification cancomprise any imaging system using ionizing or non-ionizing radiation.The term imaging system as used throughout the specification cancomprise any imaging system utilizing x-rays, e.g. a radiography system,a projection radiography system, a fluoroscopy system, a 2D fluoroscopysystem, a 3D fluoroscopy system (e.g. using a 3D C-arm system), a conebeam CT system, a spiral CT system, CT imaging systems using pencil beamgeometry, fan beam geometry, open beam geometry, a CT imaging systemusing a single detector array, a CT imaging system using multipledetector arrays, an electron beam CT imaging system, a conventionalradiography system, a digital radiography system, a digitaltomosynthesis system, a dual energy imaging system, a dual energysubtraction imaging system, a subtraction imaging system, an angiographyimaging system, a uni-planar angiography system, a bi-planar angiographysystem, a 3D angiography system; the term imaging system as usedthroughout the specification can comprise a magnetic resonance imaging(MRI) system, an ultrasound imaging system; the term imaging system asused throughout the specification can comprise a radionuclide imagingsystem, a scintillation detector imaging system for radionuclideimaging, a semiconductor detector imaging system for radionuclideimaging, a pulse height spectroscopy imaging system for radionuclideimaging, a planar nuclear imaging system, a cardiac radionuclide imagingsystem, a single photon emission computed tomography (SPECT) imagingsystem, a positron emission tomography (PET) imaging system. The termimaging system as used throughout the specification can comprise or anycombination of the foregoing imaging systems, e.g. a combinedx-ray—ultrasound imaging system, a SPECT MRI imaging system, a PET MRIimaging system, a 2D radiography/fluoroscopy— 3D cone beam CT imagingsystem etc.

In any of the embodiments, the term imaging parameter can be usedinterchangeably with the terms image acquisition parameter, acquisitionparameter, acquisition setting, image acquisition setting.

TABLE 1 Non-limiting examples of imaging parameters, settings,geometries (partial list) of one or more components of an imagingsystem, available for setting, adjustment, modification for differentimaging systems, e.g. using a virtual user interface displayed by one ormore head mounted displays or other augmented reality display systems:X-ray, fluoroscopy, C-arm (2D, 3D imaging, e.g. including cone beam CT),CT (e.g. for different scanners, geometries, CT technologies (forexample spiral CT, etc.) described in specification or known in theart): x-ray tube setting kVp mAs collimation tube - detector distancetube - patient distance patient - detector distance patient - imageintensifier distance table height (e.g. relative to tube, detector, orcombination thereof) table position (e.g. relative to tube, detector, orcombination thereof) patient position C-arm position, orientation,location gantry position, orientation, location collimation grid heightgrid width grid ratio field of view center of a field of view margins,perimeter, limits of a field of view matrix pixel size voxel size imagesize image volume imaging plane image dimensions in x, y, z and/oroblique directions, e.g. scan coverage image location image volumelocation pitch in plane resolution slice thickness increment detectorconfiguration detector resolution detector density tube current tubepotential reconstruction algorithm, e.g. brain, soft-tissue, abdomen,bone scan range, scan boundary scan limit scan range rotational center(e.g. of a spiral acquisition, a detector movement, a tube movement, aC-arm movement) rotational axis (e.g. of a spiral acquisition, adetector movement, a tube movement, a C-arm movement) reconstructedslice thickness segmentation algorithm window and/or level brightnesscontrast display resolution Ultrasound: wavelength frequency speed speedof sound depth gain frame rate width line density persistencesensitivity dynamic range relative intensity (e.g. in dB) relativepressure (e.g. in dB) amplitude pressure amplitude ratio intensity ratiopower transducer geometry linear/phased transducer activation near fieldgeometry far field geometry beam geometry beam divergence transmitfocusing receive focusing acoustic lens geometry (e.g. deformable)pre-amplification settings beam steering settings dynamic focusingsettings signal summation settings time gain compensation settingslogarithmic compression settings demodulation, envelope detectionsettings A-mode settings B-mode settings M-mode settings Dopplersettings Doppler wave settings Doppler angle Doppler shift Doppler shiftsettings Respiratory gating Cardiac gating Respiratory gating settingsCardiac gating settings Harmonic imaging settings Harmonic wave settingsAliasing, anti-aliasing settings patient position transducerposition/orientation patient scan region position/orientation field ofview matrix pixel size voxel size image size image volume imaging planein plane resolution slice thickness scan range reconstructed slicethickness segmentation algorithm window and/or level brightness contrastdisplay resolution MRI: repetition time echo time inversion time flipangle echo train length number of excitations pulse sequence, e.g. spinecho, gradient echo, fast field echo, turbo spin echo, table heighttable position patient position field of view matrix pixel size voxelsize image size image volume imaging plane image dimensions in x, y, zand/or oblique directions, e.g. scan coverage in plane resolution slicethickness 2D Fourier Transform acquisition parameters 3D FourierTransform acquisition parameters magnetization transfer contrastparameters k-space sampling parameters coil coil parameters coilgeometry phased array coil system transmit and/or receive coil coilsensitivity coil sensitivity profile gradient coil settings gradientrise time gradient strength reconstruction algorithm scan rangereconstructed slice thickness segmentation algorithm window and/or levelbrightness contrast display resolution Radionuclide based imaging, e.g.nuclear scintigraphy, SPECT, PET etc.: table height table positionpatient position gantry position, orientation, location detectordimensions detector resolution image resolution scatter collimationcollimator settings position of septal collimators, collimator ringssingle energy acquisition settings multi energy acquisition settings 2Dacquisition settings 3D acquisition settings field of view matrix pixelsize voxel size image size image volume imaging plane image dimensionsin x, y, z and/or oblique directions, e.g. scan coverage in planeresolution slice thickness reconstruction algorithm filter parametersfilter kernel radionuclide decay scan range reconstructed slicethickness segmentation algorithm window and/or level brightness contrastdisplay resolution

The term imaging parameter or imaging parameters as used throughout thespecification can comprise one or more of the above parameters and/orother parameters known in the art. Any of the foregoing parameters canbe set, defined, determined, adjusted, modified using a user interface,including a graphical user interface. The graphical user interface cancomprise virtual user interface, e.g. using a head mounted display orother augmented reality display device. The virtual interface can, forexample, use a collision detection, e.g. for generating and/or enablingone or more commands. Any of the above imaging parameters and/or otherimaging parameters known in the art can be set, defined, determined,adjusted, modified using a virtual user interface, using any of theembodiments described in the specification, and/or any combination ofembodiments described in the specification.

The term virtual interface can be used interchangeably with the termvirtual user interface. A virtual user interface can be a graphical userinterface. A virtual user interface can be displayed by one or more headmounted displays or other augmented reality display devices. The virtualuser interface displayed by a first, second, third, fourth etc. headmounted display can be the same or different. A virtual user interfacecan comprise at least one virtual object. A virtual user interface orthe at least one virtual object can comprise one or more virtual button,virtual field, virtual cursor, virtual pointer, virtual slider, virtualtrackball, virtual node, virtual numeric display, virtual touchpad,virtual keyboard, or a combination thereof. One or more commands can begenerated by an interaction, e.g. of a user, with a virtual userinterface. The interaction can be, for example, a collision detection,e.g. between a user's finger and a virtual object, e.g. a virtualbutton, or between a tracked pointer or tool or surgical instrument anda virtual object.

In one embodiment, as shown in non-limiting, strictly exemplary fashionin FIG. 1, the system can comprise, for example, a tracking camera ortracking system 1170 (e.g. an optical tracking system, anelectromagnetic tracking system, one or more visible light and/orinfrared cameras, video systems, scanners, e.g. 3D scanners, laserscanners, LIDAR systems, depth sensors); a server/controller/computingunit 1180; a tracker controller 1190 to receive tracking data, e.g. formarkers detected by camera 1170, optionally in real-time. Theserver/controller/computer unit can process and/or comprise current posedata, for example for detected markers, e.g. on a patient, e.g. a spinalclamp, one or more tools or instruments, and/or optionally one or moreHMDs 1200. A transform manager can convert tracked items, for examplefrom a local coordinate system (CS) to a camera coordinate system (CS)1210. A registration and/or calibration 1220 can transform a localcoordinate system to a marker coordinate system, optionally. Steps,processes, commands and/or functions 1190, 1200, 1210, 1220 can beprocessed or operated by the server/controller/computing unit 1180. Thecurrent pose data 1230 for tracked items, e.g. a patient, an anatomicstructure, a marker (for example on a spinal clamp, and/or any markerattached to a patient, and/or any marker on any tool or instrument,and/or any marker on one or more HMDs or other augmented reality displaydevices), a physical tool or instrument, and/or one or more HMDs orother augmented reality display devices including information about theviewing direction of the one or more HMDs or other augmented realitydisplay devices, can be located in the camera coordinate system and canbe transferred to one or more HMDs or other augmented reality displaydevices, e.g. a first, second, third, fourth etc. HMD 1240 or otheraugmented reality display devices. The one or more HMDs 1240 or otheraugmented reality display devices can, for example, run a Unity app andrender the one or more tracked items in the viewing perspective of therespective one or more HMDs or other augmented reality display devices.

In some embodiments, a first computing system comprising one or moreprocessor can be configured to transmit data to a second computingsystem configured to generate a display by a head mounted display orother augmented reality display device based on the transmitted data. Insome embodiments, a first computing system comprising one or moreprocessor can be configured to transmit data to a second computingsystem configured to generate a display by a head mounted display orother augmented reality display device. In some embodiments, a secondcomputing system configured to generate a display by a head mounteddisplay or other augmented reality display device can be configured totransmit data to a first computing system separate from the head mounteddisplay or other augmented reality display device. In some embodiments,the first computing system and the second computing system can beconfigured to transmit and/or receive data from each other, e.g. forupdating a display by a head mounted display or other augmented realitydisplay device.

The data or data packets that can be received and/or transmitted cancomprise, for example, any of the data listed in Table 2:

TABLE 2 Data or data packets for wireless transmission and/or receptionbetween two or more computing systems, including computing systemscommunicably connected to one, two or more mobile HMD units and/or otheraugmented reality display devices and/or computing systems communicablyconnected to a surgical robot and/or an imaging system. 2D imaging data(including data derived from 2D imaging studies) of a patient, e.g. ofan anatomic structre, at least a portion of a spine, spinal structure,joint, tooth, dental structure, vascular structure, or other body partof the patient, e.g. Pre-operative imaging data (CT, MRI, ultrasoundetc.) Intra-operative imaging data, for example received from anintra-operative imaging system, such as a 2D, 3D C-arm, cone beam CT, CTetc. 3D imaging data (including data derived from 2D and/or 3D imagingstudies) of at least a patient, e.g. of an anatomic structure, a portionof a spine, spinal structure, joint, tooth, dental structure, vascularstructure, or other body part of the patient Pre-operative imaging data(CT, MRI, ultrasound etc.) Intra-operative imaging data, for examplereceived from an intra-operative imaging system, such as a 2D, 3D C-arm,cone beam CT, CT etc. Coordinate data or information of a patient, e.g.of an anatomic structure, a spine, spinal structure, joint, tooth,dental structure, vascular structure, or other body part of the patientReal-time or near real-time tracking data or information of a patient,e.g. of an anatomic structure, a spine, spinal structure, joint, tooth,dental structure, vascular structure, or other body part of the patientCoordinate data or information of one or more physical pointer Real-timeor near real-time tracking data or information of one or more physicalpointer Coordinate data or information of one or more physical tools orinstruments Real-time or near real-time tracking data or information ofone or more physical tools or instruments Coordinate information of oneor more physical implants, physical implant components, or physicaltrial implants Real-time or near real-time tracking data or informationof one or more physical implants, physical implant components, orphysical trial implants Coordinate data or information of one or moreend effectors, physical tools or instruments integrated or attached toor part of a robot, e.g. a robotic arm, handheld robot or a combinationthereof, and/or coordinate data or information of one or more componentsof the robot, for example with at least a portion of the coordinate dataor information generated with use of pose data, sensor data, cameradata, 3D scanner data, controller data, drive data, actuator data, endeffector data or a combination thereof of the robot (e.g. intrinsicdata) (for example obtained using internal or integrated sensors,potentiometers, cameras, video systems, 3D scanners, LIDAR systems,depth sensors, inertial measurement units, accelerometers,magnetometers, gyroscopes, force sensors, pressure sensors, positionsensors, orientation sensors, motion sensors, position and/ororientation feedback from step motors, position and/or orientationfeedback from electric motors, position and/or orientation feedback fromhydraulic motors, position and/or orientation feedback from electricand/or mechanical actuators, position and/or orientation feedback fromdrives, position and/or orientation feedback from robotic controllers,position and/or orientation feedback from one or more robotic computerprocessors, or a combination thereof) Real-time or near real-timetracking data or information of one or more end effectors, physicaltools or instruments integrated or attached to or part of a robot, e.g.a robotic arm, handheld robot or a combination thereof, and/or real-timeor near real-time tracking data or information of one or more componentsof the robot, for example with at least a portion of the tracking dataor information generated with use of pose data, sensor data, cameradata, 3D scanner data, controller data, drive data, actuator data, endeffector data or a combination thereof of the robot (e.g. intrinsicdata) (for example obtained using internal or integrated sensors,potentiometers, cameras, video systems, 3D scanners, LIDAR systems,depth sensors, inertial measurement units, accelerometers,magnetometers, gyroscopes, force sensors, pressure sensors, positionsensors, orientation sensors, motion sensors, position and/ororientation feedback from step motors, position and/or orientationfeedback from electric motors, position and/or orientation feedback fromhydraulic motors, position and/or orientation feedback from electricand/or mechanical actuators, position and/or orientation feedback fromdrives, position and/or orientation feedback from robotic controllers,position and/or orientation feedback from one or more robotic computerprocessors, or a combination thereof) Coordinate information of one ormore physical implants, physical implant components, or physical trialimplants attached to a robot, e.g. a robotic arm, handheld robot or acombination thereof, for example with at least a portion of thecoordinate data or information generated with use of pose data, sensordata, camera data, 3D scanner data, controller data, drive data,actuator data, end effector data or a combination thereof of the robot(e.g. intrinsic data) (for example obtained using internal or integratedsensors, potentiometers, cameras, video systems, 3D scanners, LIDARsystems, depth sensors, inertial measurement units, accelerometers,magnetometers, gyroscopes, force sensors, pressure sensors, positionsensors, orientation sensors, motion sensors, position and/ororientation feedback from step motors, position and/or orientationfeedback from electric motors, position and/or orientation feedback fromhydraulic motors, position and/or orientation feedback from electricand/or mechanical actuators, position and/or orientation feedback fromdrives, position and/or orientation feedback from robotic controllers,position and/or orientation feedback from one or more robotic computerprocessors, or a combination thereof) Real-time or near real-timetracking data or information of one or more physical implants, physicalimplant components, or physical trial implants attached to a robot, e.g.a robotic arm, handheld robot or a combination thereof, for example withat least a portion of the tracking data or information generated withuse of pose data, sensor data, camera data, 3D scanner data, controllerdata, drive data, actuator data, end effector data or a combinationthereof of the robot (e.g. intrinsic data) (for example obtained usinginternal or integrated sensors, potentiometers, cameras, video systems,3D scanners, LIDAR systems, depth sensors, inertial measurement units,accelerometers, magnetometers, gyroscopes, force sensors, pressuresensors, position sensors, orientation sensors, motion sensors, positionand/or orientation feedback from step motors, position and/ororientation feedback from electric motors, position and/or orientationfeedback from hydraulic motors, position and/or orientation feedbackfrom electric and/or mechanical actuators, position and/or orientationfeedback from drives, position and/or orientation feedback from roboticcontrollers, position and/or orientation feedback from one or morerobotic computer processors, or a combination thereof) Coordinate dataor information of one or more components of an imaging system, forexample with at least a portion of the coordinate data or informationgenerated with use of pose data, sensor data, camera data, 3D scannerdata, controller data, drive data, actuator data, or a combinationthereof of the one or more components of the imaging system (e.g.intrinsic data) (for example obtained using internal or integratedsensors, potentiometers, cameras, video systems, 3D scanners, LIDARsystems, depth sensors, inertial measurement units, accelerometers,magnetometers, gyroscopes, force sensors, pressure sensors, positionsensors, orientation sensors, motion sensors, position and/ororientation feedback from step motors, position and/or orientationfeedback from electric motors, position and/or orientation feedback fromhydraulic motors, position and/or orientation feedback from electricand/or mechanical actuators, position and/or orientation feedback fromdrives, position and/or orientation feedback from one or morecontrollers, position and/or orientation feedback from one or morecomputer processors, or a combination thereof) Real-time or nearreal-time tracking data or information of one or more components of animaging system, for example with at least a portion of the tracking dataor information generated with use of pose data, sensor data, cameradata, 3D scanner data, controller data, drive data, actuator data, or acombination thereof of the one or more components of an imaging system(e.g. intrinsic data) (for example obtained using internal or integratedsensors, potentiometers, cameras, video systems, 3D scanners, LIDARsystems, depth sensors, inertial measurement units, accelerometers,magnetometers, gyroscopes, force sensors, pressure sensors, positionsensors, orientation sensors, motion sensors, position and/ororientation feedback from step motors, position and/or orientationfeedback from electric motors, position and/or orientation feedback fromhydraulic motors, position and/or orientation feedback from electricand/or mechanical actuators, position and/or orientation feedback fromdrives, position and/or orientation feedback from one or morecontrollers, position and/or orientation feedback from one or morecomputer processors, or a combination thereof) Coordinate data orinformation of one or more end effectors, physical tools or instrumentsintegrated or attached to or part of a robot, e.g. a robotic arm,handheld robot or a combination thereof, and/or coordinate data orinformation of one or more components of the robot, for example with atleast a portion of the coordinate data or information generated with useof one or more cameras, video systems, 3D scanners, LIDAR systems, depthsensors, or combination thereof, integrated or attached to one or moreHMDs or other augmented reality display systems, separate from one ormore HMDs or other augmented reality display systems (e.g. on a stand,tripod, attached to or integrated into OR lighting, OR fixtures, animaging system (e.g. x-ray, cone beam CT, CT)), or a combinationthereof, with one or more computer processors configured, using the oneor more cameras, video systems, 3D scanners, LIDAR systems, depthsensors, or combination thereof, to determine, for example, theposition, orientation, direction of movement, one or more coordinates,or combination thereof of the one or more end effectors, physical toolsor instruments integrated or attached to or part of a robot, and/or oneor more markers, e.g. active markers (e.g. RF markers), passive markers(e.g. infrared markers), optical markers (e.g. with geometric patterns,QR codes, bar codes, defined shapes, e.g. triangles, squares, rectanglesetc.), LEDs or a combination thereof integrated or attached to the oneor more physical tools or instruments, integrated or attached to atleast portions of the robot (e.g. the robotic arm, handheld robot or thecombination thereof), or integrated or attached to the one or morephysical tools or instruments and integrated or attached to at leastportions of the robot Real-time or near real-time tracking data orinformation of one or more end effectors, physical tools or instrumentsintegrated or attached to a robot, e.g. a robotic arm, handheld robot ora combination thereof, and/or real-time or near real-time tracking dataor information of one or more components of the robot, for example withat least a portion of the tracking data or information generated withuse of one or more cameras, video systems, 3D scanners, LIDAR systems,depth sensors, or combination thereof, integrated or attached to one ormore HMDs or other augmented reality display systems, separate from oneor more HMDs or other augmented reality display systems (e.g. on astand, tripod, attached to or integrated into OR lighting, OR fixtures,an imaging system (e.g. x-ray, cone beam CT, CT)), or a combinationthereof, with one or more computer processors configured, using the oneor more cameras, video systems, 3D scanners, LIDAR systems, depthsensors, or combination thereof, to determine, for example, theposition, orientation, direction of movement, one or more coordinates,or combination thereof of the one or more end effectors, physical toolsor instruments integrated or attached to or part of a robot, and/or oneor more markers, e.g. active markers (e.g. RF markers), passive markers(e.g. infrared markers), optical markers (e.g. with geometric patterns,QR codes, bar codes, defined shapes, e.g. triangles, squares, rectanglesetc.), LEDs or a combination thereof integrated or attached to the oneor more physical tools or instruments, integrated or attached to atleast portions of the robot (e.g. the robotic arm, handheld robot or thecombination thereof), or integrated or attached to the one or morephysical tools or instruments and integrated or attached to at leastportions of the robot Coordinate information of one or more physicalimplants, physical implant components, or physical trial implantsattached to a robot, e.g. a robotic arm, handheld robot or a combinationthereof, for example with at least a portion of the coordinate data orinformation generated with use of one or more cameras, video systems, 3Dscanners, LIDAR systems, depth sensors, or combination thereof,integrated or attached to one or more HMDs or other augmented realitydisplay systems, separate from one or more HMDs or other augmentedreality display systems (e.g. on a stand, tripod, attached to orintegrated into OR lighting, OR fixtures, an imaging system (e.g. x-ray,cone beam CT, CT)), or a combination thereof, with one or more computerprocessors configured, using the one or more cameras, video systems, 3Dscanners, LIDAR systems, depth sensors, or combination thereof, todetermine, for example, the position, orientation, direction ofmovement, one or more coordinates, or combination thereof of the one ormore physical implants, physical implant components, or physical trialimplants attached to a robot, and/or one or more markers, e.g. activemarkers (e.g. RF markers), passive markers (e.g. infrared markers),optical markers (e.g. with geometric patterns, QR codes, bar codes,defined shapes, e.g. triangles, squares, rectangles etc.), LEDs or acombination thereof integrated or attached to the one or more physicaltools or instruments, integrated or attached to at least portions of therobot (e.g. the robotic arm, handheld robot or the combination thereof),or integrated or attached to the one or more physical tools orinstruments and integrated or attached to at least portions of the robotReal-time or near real-time tracking data or information of one or morephysical implants, physical implant components, or physical trialimplants attached to a robot, e.g. a robotic arm, handheld robot or acombination thereof, for example with at least a portion of the trackingdata or information generated with use of one or more cameras, videosystems, 3D scanners, LIDAR systems, depth sensors, or combinationthereof, integrated or attached to one or more HMDs or other augmentedreality display systems, separate from one or more HMDs or otheraugmented reality display systems (e.g. on a stand, tripod, attached toor integrated into OR lighting, OR fixtures, an imaging system (e.g.x-ray, cone beam CT, CT)), or a combination thereof, with one or morecomputer processors configured, using the one or more cameras, videosystems, 3D scanners, LIDAR systems, depth sensors, or combinationthereof, to determine, for example, the position, orientation, directionof movement, one or more coordinates, or combination thereof of the oneor more physical implants, physical implant components, or physicaltrial implants attached to a robot, and/or one or more markers, e.g.active markers (e.g. RF markers), passive markers (e.g. infraredmarkers), optical markers (e.g. with geometric patterns, QR codes, barcodes, defined shapes, e.g. triangles, squares, rectangles etc.), LEDsor a combination thereof integrated or attached to the one or morephysical tools or instruments, integrated or attached to at leastportions of the robot (e.g. the robotic arm, handheld robot or thecombination thereof), or integrated or attached to the one or morephysical tools or instruments and integrated or attached to at leastportions of the robot Coordinate data or information of one or morecomponents of an imaging system, for example with at least a portion ofthe coordinate data or information generated with use of one or morecameras, video systems, 3D scanners, LIDAR systems, depth sensors, orcombination thereof, integrated or attached to one or more HMDs or otheraugmented reality display systems, separate from one or more HMDs orother augmented reality display systems (e.g. on a stand, tripod,attached to or integrated into OR lighting, OR fixtures, the imagingsystem (e.g. x-ray, cone beam CT, CT)), or a combination thereof, withone or more computer processors configured, using the one or morecameras, video systems, 3D scanners, LIDAR systems, depth sensors, orcombination thereof, to determine, for example, the position,orientation, direction of movement, one or more coordinates, orcombination thereof of the one or more components of the imaging system,and/or one or more markers, e.g. active markers (e.g. RF markers),passive markers (e.g. infrared markers), optical markers (e.g. withgeometric patterns, QR codes, bar codes, defined shapes, e.g. triangles,squares, rectangles etc.), LEDs or a combination thereof integrated orattached to the one or more components of the imaging system Real-timeor near real-time tracking data or information of one or more componentsof an imaging system, for example with at least a portion of thetracking data or information generated with use of one or more cameras,video systems, 3D scanners, LIDAR systems, depth sensors, or combinationthereof, integrated or attached to one or more HMDs or other augmentedreality display systems, separate from one or more HMDs or otheraugmented reality display systems (e.g. on a stand, tripod, attached toor integrated into OR lighting, OR fixtures, the imaging system (e.g.x-ray, cone beam CT, CT)), or a combination thereof, with one or morecomputer processors configured, using the one or more cameras, videosystems, 3D scanners, LIDAR systems, depth sensors, or combinationthereof, to determine, for example, the position, orientation, directionof movement, one or more coordinates, or combination thereof of the oneor more components of the imaging system Coordinate data or informationof one or more HMDs or other augmented reality display systems(including, for example, but not limited to, HMD housing, HMD visor, HMDdisplay [e.g. a mirror, combiner, optical waveguide, optics, displayunit]), comprising, for example, information about HMD or otheraugmented reality display system position, orientation, pitch or tilt,direction of movement, optionally labeled or coded for each specific HMDor other augmented reality display system (e.g. by a computer processorintegrated into, attached to, or connected to a camera or scanner, acomputer processor integrated into, attached to, or connected to a firstcomputing unit (e.g. in a server), and/or a computer processorintegrated into, attached to, or connected to a second computing unit(e.g. in a client, for example integrated into an HMD or other augmentedreality display system or connected to an HMD or other augmented realitydisplay system) Real-time or near real-time tracking data of one or moreHMDs or other augmented reality display systems (including, for example,but not limited to, HMD housing, HMD visor, HMD display [e.g. a mirror,combiner, optical waveguide, optics, display unit]), comprising, forexample, information about HMD or other augmented reality display systemposition, orientation, pitch or tilt, direction of movement, optionallylabeled or coded for each specific HMD or other augmented realitydisplay system (e.g. by a computer processor integrated into, attachedto, or connected to a camera or scanner, a computer processor integratedinto, attached to, or connected to a first computing unit (e.g. in aserver), and/or a computer processor integrated into, attached to, orconnected to a second computing unit (e.g. in a client, for exampleintegrated into an HMD or other augmented reality display system orconnected to an HMD or other augmented reality display system) Real-timeor near real-time data about the position, orientation, position andorientation, pitch or tilt, direction of movement of one or more HMDs orother augmented reality display system (including, for example, but notlimited to, HMD housing, HMD visor, HMD display [e.g. a mirror,combiner, optical waveguide, optics, display unit]), optionally labeledor coded for each specific HMD or other augmented reality display system(e.g. by a computer processor integrated into, attached to, or connectedto a camera or scanner, a computer processor integrated into, attachedto, or connected to a first computing unit (e.g. in a server), and/or acomputer processor integrated into, attached to, or connected to asecond computing unit (e.g. in a client, for example integrated into anHMD or other augmented reality display system or connected to an HMD orother augmented reality display system) User specific settings and/orparameters User specific display settings (e.g. color preferences,location of alphanumeric data, scan data, image data, 3D displays withinthe field of view of an HMD) Interpupillary distance, e.g. determinedvia physical measurement and/or electronic measurement (for example bymoving/aligning virtual objects displayed by an HMD); optionally storedon a first computing system, e.g. communicably coupled to a server, forexample connected to a navigation system, robot, and/or imaging system,and transmitted to/received by a second computing system communicablycoupled to the HMD; or optionally stored on a second computing systemcommunicably coupled to the HMD and transmitted to/received by a firstcomputing system, e.g. communicably coupled to a server, for exampleconnected to a navigation system, robot, and/or imaging systemCoordinate data or information of one or more virtual display(including, for example, a virtual retinal display) (e.g. a virtualinterface) by one or more HMDs or other augmented reality displaysystems, comprising, for example, information about virtual displayposition, orientation, pitch or tilt, direction of movement, optionallylabeled or coded for each specific virtual display and/or HMD or otheraugmented reality display system (e.g. by a computer processorintegrated into, attached to, or connected to a camera or scanner, acomputer processor integrated into, attached to, or connected to a firstcomputing unit (e.g. in a server), and/or a computer processorintegrated into, attached to, or connected to a second computing unit(e.g. in a client, for example integrated into an HMD or other augmentedreality display system or connected to an HMD or other augmented realitydisplay system) Real-time or near real-time tracking data of one or morevirtual display (including, for example, a virtual retinal display)(e.g. a virtual interface) by one or more HMDs or other augmentedreality display systems, comprising, for example, information aboutvirtual display position, orientation, pitch or tilt, direction ofmovement, optionally labeled or coded for each specific virtual displayand/or HMD or other augmented reality display system (e.g. by a computerprocessor integrated into, attached to, or connected to a camera orscanner, a computer processor integrated into, attached to, or connectedto a first computing unit (e.g. in a server), and/or a computerprocessor integrated into, attached to, or connected to a secondcomputing unit (e.g. in a client, for example integrated into an HMD orother augmented reality display system or connected to an HMD or otheraugmented reality display system) Real-time or near real-time data aboutthe position, orientation, position and orientation, pitch or tilt,direction of movement of one or more virtual display (including, forexample, a virtual retinal display) (e.g. a virtual interface) by one ormore HMDs or other augmented reality display system, optionally labeledor coded for each specific virtual display and/or HMD or other augmentedreality display system (e.g. by a computer processor integrated into,attached to, or connected to a camera or scanner, a computer processorintegrated into, attached to, or connected to a first computing unit(e.g. in a server), and/or a computer processor integrated into,attached to, or connected to a second computing unit (e.g. in a client,for example integrated into an HMD or other augmented reality displaysystem or connected to an HMD or other augmented reality display system)3D volume information of one or more physical tools or instrumentsReal-time or near real-time data about one or more interactions, e.g.one or more collisions, of a tracked physical tool or instrument, e.g. atracked pointer, a tracked stylus, a tracked tool, a tracked instrumentor a combination thereof, with one or more virtual displays (e.g. avirtual interface) (including, for example, a virtual retinal display)Real-time or near real-time changes, modifications, or alterations ofone or more virtual displays (e.g. a virtual interface) (including, forexample, a virtual retinal display) as a result of or triggered by oneor more interactions, e.g. one or more collisions, with a trackedphysical tool or instrument, e.g. a tracked pointer, a tracked stylus, atracked tool, a tracked instrument or a combination thereof 3D volumeinformation of one or more physical implants, physical implantcomponents, or physical trial implants 3D volume information of one ormore physical tools or instruments 3D surface information of one or morephysical tools or instruments 3D surface information of one or morephysical implants, physical implant components, or physical trialimplants 2D or 3D representations (for example 2D or 3D anatomic models,models derived from imaging data, e.g. of a patient) (including, forexample, 3D volume or 3D surface data) of at least a portion of one ormore structures or body parts of a patient e.g. of at least a portion ofa spine, spinal structure, joint, tooth, dental structure, vascularstructure, or other body part 2D or 3D representations (including, forexample, 3D volume or 3D surface data) of at least a portion of one ormore virtual tools or instruments (e.g. corresponding to at least aportion of one or more physical tools or instruments, e.g. a tool orinstrument axis) 2D or 3D representations (including, for example, 3Dvolume or 3D surface data) of at least a portion of one or more virtualimplants, virtual implant components, or virtual trial implants (e.g.corresponding to at least a portion of one or more physical implants,physical implant components, or physical trial implants, e.g. an implantaxis and/or an implant outline) 2D or 3D representations (including, forexample, 3D volume or 3D surface data) of virtual surgical guides, e.g.one or more virtual lines, virtual trajectories, virtual axes, virtualplanes, virtual cut planes, virtual saw blades, virtual cut blocks,virtual inserts etc. Target data, targeting data, e.g. in 2D or 3DStereoscopic view of any of the foregoing, e.g. generated for one ormore HMDs (for example accounting for user specification and/orcharacteristics, such as interpupillary distance Any combination of oneor more of the foregoing

The data or data packets can comprise stereoscopic and/ornon-stereoscopic views or data prepared for stereoscopic and/ornon-stereoscopic views or displays by one or more HMDs or otheraugmented reality display systems. Stereoscopic and/or non-stereoscopicviews or data prepared for stereoscopic and/or non-stereoscopic views ordisplays by one or more HMDs or other augmented reality display systemscan be updated in near real-time or real-time, e.g. with a rate of 10Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz, 100 Hz, or any other rate or frequencyincluding higher rates or frequencies.

Position, orientation, position and orientation, direction of movementand/or tracking data for one or more HMDs or other augmented realitydisplay systems can be obtained, measured and/or generated using, forexample, one or more cameras or scanners (e.g. video systems, 3Dscanners, LIDAR systems, depth sensors; using visible light and/orinfrared and/or any other wavelength) integrated into or attached to theone or more HMDs or other augmented reality display systems, one or morecameras and/or scanners (e.g. video systems, 3D scanners, LIDAR systems,depth sensors; using visible light and/or infrared and/or any otherwavelength) separate from the one or more HMDs or other augmentedreality display systems, one or more lasers, e.g. integrated or attachedto the one or more HMDs or other augmented reality display systemsand/or separate from the one or more HMDs or other augmented realitydisplay systems, one or more inertial measurement units integrated orattached to the one or more HMDs or other augmented reality displaysystems or a surgeon's head, one or more markers, e.g. active markers(e.g. RF markers), passive markers (e.g. infrared markers), opticalmarkers (e.g. with geometric patterns, QR codes, bar codes, definedshapes, e.g. triangles, squares, rectangles etc.) integrated or attachedto the one or more HMDs or other augmented reality display systems, anycombination thereof optionally visible to the camera(s) and/orscanner(s), or any combination thereof.

Position, orientation, position and orientation, direction of movementand/or tracking data for one or more HMDs or other augmented realitydisplay systems can be generated, transmitted and/or received in nearreal time or in real time, e.g. with a rate of 10 Hz, 15 Hz, 20 Hz, 25Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80Hz, 90 Hz, 100 Hz, or any other rate or frequency including higher ratesor frequencies.

Position, orientation, position and orientation, direction of movementand/or tracking data for one or more HMDs or other augmented realitydisplay systems can be transmitted by one or more computer processorsintegrated into or connected to the one or more HMDs or other augmentedreality display systems via a wireless access point or router wirelesslyto a separate computing system with one or more computer processors. Theseparate computing system can process the data about the position,orientation, position and orientation, direction of movement and/ortracking data of the one or more HMDs or other augmented reality displaysystems received and package them with other data for corresponding timepoints or time intervals, e.g. patient tracking data and/or instrumenttracking data, for transmission, optionally back to the one or more HMDsor other augmented reality display systems.

Position, orientation, position and orientation, direction of movementand/or tracking data for one or more HMDs or other augmented realitydisplay systems can be obtained, acquired and/or generated by one ormore cameras or scanners separate from the one or more HMDs or otheraugmented reality display systems and can be processed by one or morecomputer processors connected to or integrated into the camera and/orscanner and/or connected to or integrated into a separate computingsystem, e.g. a server, optionally connected directly or wirelessly tothe camera or scanner. The separate computing system can process thedata about the position, orientation, position and orientation,direction of movement and/or tracking data of the one or more HMDs orother augmented reality display systems received and package them withother data for corresponding time points or time intervals, e.g. patienttracking data and/or instrument tracking data, for transmission (and/orreception), for example back to the one or more HMDs or other augmentedreality display systems.

Any of the data listed in Table 2 and any additional data can betransmitted and/or received in real-time, or near real-time, withtransmitting and/or receiving rates of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90Hz, 100 Hz, or any other rate or frequency including higher rates orfrequencies.

When multiple data sets, e.g. different types of data such as instrumenttracking data and/or HMD or other augmented reality display systemtracking data and/or patient or surgical site (e.g. a spine, joint,tooth or vascular structure) tracking data and/or virtual user interfaceand/or interaction with virtual user interface data (e.g. with a trackedphysical tool or instrument) are transmitted and/or received, they canbe transmitted and/or received simultaneously or non-simultaneously.Data sets including any of the data listed in Table 2 can optionally belabelled, e.g. with a time stamp, time point, time interval (e.g. within1 transmission or data reception, for example, for a rate of 60 Hz,within 16.66 ms or less or, for example, any other value within the timeallocated for transmission and reception), a time label, a time tag orany combination thereof.

In some embodiments, coordinate information, registration data, trackingdata or a combination thereof of one or more HMDs or other augmentedreality display systems can optionally be labeled or coded for eachspecific HMD or other augmented reality display system, e.g. by acomputer processor integrated into, attached to, or connected to acamera or scanner (optionally part of a first or second computing unit),a computer processor integrated into, attached to, or connected to afirst computing unit (e.g. in a server), and/or a computer processorintegrated into, attached to, or connected to a second computing unit(e.g. in a client, for example integrated into an HMD or other augmentedreality display system or connected to an HMD or other augmented realitydisplay system).

In some embodiments, data packets (for example, as listed in Table 2)can comprise multiple types of data, e.g. data comprising instrumenttracking data, data comprising HMD or other augmented reality displaysystem tracking data and/or a data comprising patient or surgical site(e.g. a spine, joint, dental, vascular, organ or neural structure)tracking data and/or virtual user interface data and/or interaction withvirtual user interface data (e.g. with a tracked physical tool orinstrument), all packaged within the same data packet. As the datapacket(s) is/are transmitted or received, e.g. data comprisinginstrument tracking data and/or data comprising HMD or other augmentedreality display system tracking data and/or a data comprising patient orsurgical site (e.g. a spine, joint, dental, vascular, organ or neuralstructure) tracking data and/or virtual user interface data and/orinteraction with virtual user interface data (e.g. with a trackedphysical tool or instrument) can be transmitted or received together,e.g. simultaneously.

In some embodiments, transmission and/or reception can be processed byone or more computer processors, e.g. in a first and/or a secondcomputing system, and/or integrated or attached to a camera or scanner(e.g. integrated or attached to an HMD or other augmented realitydisplay system, integrated or attached to a robot, separate from an HMDor other augmented reality display system or robot etc.), so that data,for example, instrument tracking data and/or HMD or other augmentedreality display system tracking data and/or patient or surgical site(e.g. a spine or joint) tracking data, acquired with the same timestamp, time point, time label, time tag, or within the same timeinterval or any combination thereof are transmitted and/or received inthe same data packet.

In some embodiments, transmission and/or reception can be processed byone or more computer processors so that data, for example, instrumenttracking data and/or HMD or other augmented reality display systemtracking data and/or patient or surgical site (e.g. a spine or joint)tracking data and/or virtual user interface data and/or interaction withvirtual user interface data (e.g. with a tracked physical tool orinstrument), acquired within the same time interval (and optionallylabelled with the same time interval) are transmitted and/or received byone or more computer processors, e.g. in a first and/or a secondcomputing system, within a defined time period in multiple data packets;the defined time period can be corresponding to, matching, oroverlapping with the time interval. Optionally the defined time periodfor transmitting and/or receiving data packets can be a time periodbounded or defined by or derived from the transmission and/or receptionrate, e.g. <0.16666666 sec for a transmission and/or reception rate of60 Hz, or <0.0333333 sec for a transmission and/or reception rate of 30Hz, <0.04 sec for a transmission and/or reception rate of 25 Hz, or anyother value.

Data packets (for example, as listed in Table 2), e.g. a first datapacket comprising instrument tracking data, a second data packetcomprising HMD or other augmented reality display system tracking dataand/or a third data packet comprising patient or surgical site (e.g. aspine, joint, dental, vascular, organ or neural structure) tracking dataand/or a fourth data packet comprising virtual user interface dataand/or interaction with virtual user interface data (e.g. with a trackedphysical tool or instrument) can be transmitted and/or receivedsimultaneously, for example using different frequencies. Data packets,e.g. a first data packet comprising instrument tracking data, a seconddata packet comprising HMD or other augmented reality display systemtracking data and/or a third data packet comprising patient or surgicalsite (e.g. a spine or joint) tracking data and/or a fourth data packetcomprising virtual user interface data and/or interaction with virtualuser interface data (e.g. with a tracked physical tool or instrument)can be transmitted and/or received sequentially (e.g. using the same ordifferent frequencies). Data packets, e.g. a first data packetcomprising instrument tracking data, a second data packet comprising HMDor other augmented reality display system tracking data and/or a thirddata packet comprising patient or surgical site (e.g. a spine or joint)tracking data and/or a fourth data packet comprising virtual userinterface data and/or interaction with virtual user interface data (e.g.with a tracked physical tool or instrument) can be transmitted and/orreceived in an offset manner, e.g. with a pause or lag interval spacedin between. Data packets, e.g. a first data packet comprising instrumenttracking data, a second data packet comprising HMD or other augmentedreality display system tracking data and/or a third data packetcomprising patient or surgical site (e.g. a spine or joint) trackingdata and/or a fourth data packet comprising virtual user interface dataand/or interaction with virtual user interface data (e.g. with a trackedphysical tool or instrument) can be transmitted and/or received in aninterleaved manner. Data packets, e.g. a first data packet comprisinginstrument tracking data, a second data packet comprising HMD or otheraugmented reality display system tracking data and/or a third datapacket comprising patient or surgical site (e.g. a spine or joint)tracking data and/or a fourth data packet comprising virtual userinterface data and/or interaction with virtual user interface data (e.g.with a tracked physical tool or instrument) can be transmitted and/orreceived in a non-overlapping manner during the transmission and/or thereception. Data packets, e.g. a first data packet comprising instrumenttracking data, a second data packet comprising HMD or other augmentedreality display system tracking data and/or a third data packetcomprising patient or surgical site (e.g. a spine or joint) trackingdata and/or a fourth data packet comprising virtual user interface dataand/or interaction with virtual user interface data (e.g. with a trackedphysical tool or instrument) can be transmitted and/or received in anoverlapping manner during the transmission and/or the reception.

Thus, data packets, e.g. comprising one or more of data comprisinginstrument tracking data, data comprising HMD or other augmented realitydisplay system tracking data, or data comprising patient or surgicalsite data can be transmitted or received in a simultaneous, synchronousfashion and/or alternatively in an non-synchronous or asynchronousfashion.

The data can be transmitted in near real time or in real time, e.g. witha rate of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz,55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz, 100 Hz, or any other rate orfrequency including higher rates or frequencies.

Data transmission can be performed using wireless data transmissionprotocols known in the art, e.g. Bluetooth, Wifi, LiFi, etc. and, asdescribed, for example in PCT International Application Serial Nos.PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/061698 andPCT/US2019/015522, which are hereby incorporated by reference in theirentirety.

In some embodiments, the system can comprise a tracking system orsensor, e.g. optical tracking systems, for example using infrared and/orvisible light cameras, video systems, 3D scanners, LIDAR systems, depthsensors, radiofrequency tracking systems, or combinations thereof, e.g.for outside-in or inside-out tracking. Multiple tracking systems can beused at the same time or, optionally combined, e.g. inside-out andoutside-in tracking. Any tracking, sensor, or registration system knownin the art (e.g. optical tracking systems, for example using infraredand/or visible light cameras, video systems, 3D scanners, LIDAR systems,depth sensors, radiofrequency tracking systems, or combinationsthereof), can be used, for example as described in a non-limitingfashion in PCT International Application Serial Nos. PCT/US2017/021859,PCT/US2018/013774, PCT/US2019/061698 and PCT/US2019/015522, which arehereby incorporated by reference in their entirety.

A first computing system can comprise one or more computer processors.The second computing system can comprise one or more computerprocessors. The second computing system can be part of one or moremobile, wireless HMD or other augmented reality display system units.The second computing system can be part of a first mobile, wireless HMDor other augmented reality display system unit. A third computing systemcan be part of a second mobile wireless HMD or other augmented realitydisplay system unit. A fourth computing system can be part of a thirdmobile wireless HMD or other augmented reality display system unit. Afifth computing system can be part of a third mobile wireless HMD orother augmented reality display system unit, etc.

A first computing system can comprise one or more computer processors. Afirst computing system can, for example, be a server or controller orcomputing unit as shown in FIG. 1, 1180. The one or more computerprocessors can be configured to run different operating modules asshown, for example, in FIG. 2, e.g.

-   -   A tracking module or tracking engine 1100. The tracking module        or tracking engine can comprise a tracking system or sensor,        e.g. a video camera, infrared camera, 3D scanner, laser scanner,        LIDAR, imaging system etc., e.g. for outside-in or inside-out        tracking. The one or more computer processors and software        running on the one or more computer processor can comprise an        interface, e.g. a graphical user interface, for operating the        tracking system or tracking sensor. An interface can be        configured for communication with other operating modules, e.g.        integrated with the first computing system or the second        computing system. The tracking module or tracking engine 1100        can, for example, obtain tracking information or data and/or        track one or more physical instruments and/or tools, one or more        anatomic structures, landmarks, and/or surfaces of a patient        (e.g. in a surgical site) and/or one or more HMDs or other        augmented reality display systems. The tracking module or engine        1100 can, optionally, label or code the tracking data and/or        information for each specific HMD or other augmented reality        display system, e.g. by a computer processor integrated into,        attached to, or connected to a camera or scanner (optionally        part of a first or second computing unit), a computer processor        integrated into, attached to, or connected to a first computing        unit (e.g. in a server), and/or a computer processor integrated        into, attached to, or connected to a second computing unit (e.g.        in a client, for example integrated into an HMD or other        augmented reality display system or connected to an HMD or other        augmented reality display system). The tracking module or engine        1100 can, for example, apply a label or code specific for each        HMD or other augmented reality display system based on tracking        data and/or information received for and/or from each individual        HMD or other augmented reality display system. The tracking data        and/or information can, for example, be data or information from        a camera or scanner integrated into or attached to an HMD or        other augmented reality display system. The tracking data and/or        information can, for example, be from a camera or scanner        separate from an HMD or other augmented reality display system.        The tracking data and/or information can, for example, be from        one or more cameras or scanners integrated into an HMD or other        augmented reality display system or attached or connected to an        HMD or other augmented reality display system and from one or        more cameras or scanners separate from an HMD or other augmented        reality display system (e.g. attached to an OR light, OR        fixture, OR wall, a robot etc.). The tracking data and/or        information can, for example, comprise data of one or more        markers, e.g. attached to one or more HMDs or other augmented        reality display systems, a patient (e.g. an anatomic structure,        landmark or surface of the patient), a physical surgical tool or        instrument, a physical implant or a combination thereof. In some        embodiments, one or more markers can be configured specific for        each HMD or other augmented reality display system, e.g. using a        unique bar code, QR code, fiducial marker configuration, RF        signal etc., which can, for example, be recognized by one or        more computer processors.    -   An instrument calibration module 1110. The one or more computer        processors can be configured to run an instrument calibration        module. The instrument calibration module can comprise an        interface. The instrument calibration module can be configured        for determining an instrument or tool tip, an instrument or tool        axis, an instrument or tool length, an instrument or tool        rotational wobble etc. The tracking module or tracking engine        and/or an interface can be configured for communication with        other operating modules, e.g. integrated with the first        computing system or the second computing system.    -   A headset calibration module 1120. The one or more computer        processors can be configured to run a headset calibration        module. The headset calibration module can comprise an        interface. The headset calibration module, including its        interface, can, for example, be configured for determining an        interpupillary distance, a distance from the eye (e.g. pupil,        iris, retina) to the display (e.g. a waveguide, mirror etc.),        and/or for setting or adjusting an interpupillary distance in        the headset, for setting or adjusting a field of view, for        setting or adjusting a distance from the eye to the display for        a given user, for setting or adjusting user display preferences        (e.g. color, targeting tools, magnification, alphanumeric        display, display window location, alphanumeric data location        etc. The headset calibration module and/or an interface can be        configured for communication with other operating modules, e.g.        integrated with the first computing system or the second        computing system.    -   An imaging and navigation module 1130. The one or more computer        processors can be configured to run an imaging and navigation        module. The imaging and navigation module can, for example,        comprise a PACS interface, an interface to an imaging system,        e.g. x-ray, C-arm, 3D C-arm, cone-beam CT, CT, MRI etc., a DICOM        reader, an image processing and display unit, e.g. for        multi-planar reconstruction and/or display, 3D reconstruction        and/or display, a planning module, e.g. with an interface for a        user, surgeon (e.g. for screw or implant selection and        placement), a navigation module, e.g. with coordinate output        from a planning module. The imaging and navigation module and/or        an interface can be configured for communication with other        operating modules, e.g. integrated with the first computing        system or the second computing system.    -   An AR or VR wireless networking module 1140. In the AR or VR        wireless networking module, one or more computer processors can        be configured for packaging of data, e.g. data listed in        Table 2. Packing of data can, optionally, comprise data        compression. Packaging of data can, optionally, comprise        segmentation or separation into different time segments. Data        packets for different time segments can, for example, be        generated at a rate of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz,        40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz,        100 Hz, or any other rate or frequency including higher rates or        frequencies. One or more computer processors can be configured        to transmit data packets using a wireless access point to a        second, third, fourth or more computing system. One or more        computer processors can be configured to receive data packets        using the wireless access point from a second, third, fourth or        more computing system. In some embodiments, multiple wireless        access points can be used. Any wireless communication protocol        known in the art, e.g. Bluetooth, WiFi, LiFi, can be used.    -   An AR or VR visualization module 1150. One or more computer        processors can be configured for generating display data for        display by one or more HMDs or other augmented reality display        systems. The display data can comprise targeting displays, e.g.        target disk like, magnified displays, color labeled displays        (e.g. red, yellow, green indicating position or alignment        accuracy or thresholds), 3D models of the patient, image slices,        3D models or 3D or 2D graphical representations of tools or        instruments (e.g. tracked physical tools or instruments)        alphanumeric displays, and/or an AR or VR enabled graphical user        interface (e.g. gesture recognition, using gesture recognition,        virtual pointers, a virtual mouse, a virtual keyboard, virtual        buttons, gaze recognition, gaze lock or a combination thereof).        An AR or VR visualization module can comprise a display of a        virtual user interface and/or a display of one or more virtual        interactions, e.g. collisions, with a virtual user interface        (e.g. with a tracked physical tool or instrument). The AR or VR        visualization and/or display module and/or an interface can be        configured for communication with other operating modules, e.g.        integrated with the first computing system or the second        computing system.    -   An AR or VR display module 1160. One or more computer processors        can be configured for generating one or more 3D stereoscopic        views, for example at a rate similar to the data transmission or        reception and/or at a different rate. The 3D stereoscopic view        can, for example, be adjusted for user specific characteristics        (e.g. interpupillary distance, distance from pupil to mirror        etc.). The 3D stereoscopic view can, for example, be adjusted        for the distance from the pupil or the display to the patient,        e.g. a surgical site, for example, in a spine, knee, hip, organ,        vessel etc.; adjustments can comprise adjustments of focal plane        or point, adjustments of scale or magnification of virtual        displays and display items, adjustments of convergence.        Adjustments can be in real-time or near real-time. Adjustments        can be at less than real-time. An AR or VR display module can        comprise a display of a virtual user interface and/or a display        of one or more virtual interactions, e.g. collisions, with a        virtual user interface (e.g. with a tracked physical tool or        instrument).

One or more of the modules 1100-1160 can be integrated or combined. Forexample, the AR visualization module 1150 can be integrated or combinedwith the AR display module 1160.

One or more of the modules 1100-1160 can be run by the same computerprocessor or the same group of computer processors. One or more of themodules 1100-1160 can be run by different computer processors.

One or more of the computer processors operating a tracking engine ortracking module 1100, one or more computer processors operating aninstrument calibration module 1110, one or more of the computerprocessors operating a headset calibration module 1120, one or more ofthe computer processors operating an imaging and navigation module 1130,one or more of the computer processors operating an AR wirelessnetworking module 1140, one or more of the computer processors operatingan AR visualization module 1150, and/or one or more of the computerprocessors operating an AR display module 1160 can be the same.

One or more of the computer processors operating a tracking engine ortracking module 1100, one or more computer processors operating aninstrument calibration module 1110, one or more of the computerprocessors operating a headset calibration module 1120, one or more ofthe computer processors operating an imaging and navigation module 1130,one or more of the computer processors operating an AR wirelessnetworking module 1140, one or more of the computer processors operatingan AR visualization module 1150, and/or one or more of the computerprocessors operating an AR display module 1160 can be different.

The first computing system can, for example, be stationary, e.g. on acart or stand. In some embodiments, the first computing system can alsobe mobile, e.g. part of a mobile, wireless HMD system or other augmentedreality display system.

A second, third, fourth, fifth or more computing systems can compriseone or more computer processors. The one or more computer processors canbe configured to run different operating modules as shown, for example,in FIG. 2, e.g.

-   -   A tracking module or tracking engine 1100. The tracking module        or tracking engine can comprise a tracking system or sensor,        e.g. a video camera, infrared camera, 3D scanner, laser scanner,        LIDAR, imaging system etc., e.g. for outside-in or inside-out        tracking. The one or more computer processors and software        running on the one or more computer processor can comprise an        interface, e.g. a graphical user interface, for operating the        tracking system or tracking sensor. An interface can be        configured for communication with other operating modules, e.g.        integrated with the first computing system or the second        computing system. The tracking module or tracking engine 1100        can, for example, obtain tracking information or data and/or        track one or more physical instruments and/or tools, one or more        anatomic structures, landmarks, and/or surfaces of a patient        (e.g. in a surgical site) and/or one or more HMDs or other        augmented reality display systems. The tracking module or engine        1100 can, optionally, label or code the tracking data and/or        information for each specific HMD or other augmented reality        display system, e.g. by a computer processor integrated into,        attached to, or connected to a camera or scanner (optionally        part of a first or second computing unit), a computer processor        integrated into, attached to, or connected to a first computing        unit (e.g. in a server), and/or a computer processor integrated        into, attached to, or connected to a second computing unit (e.g.        in a client, for example integrated into an HMD or other        augmented reality display system or connected to an HMD or other        augmented reality display system). The tracking module or engine        1100 can, for example, apply a label or code specific for each        HMD or other augmented reality display system based on tracking        data and/or information received for and/or from each individual        HMD or other augmented reality display system. The tracking data        and/or information can, for example, be data or information from        a camera or scanner integrated into or attached to an HMD or        other augmented reality display system. The tracking data and/or        information can, for example, be from a camera or scanner        separate from an HMD or other augmented reality display system.        The tracking data and/or information can, for example, be from        one or more cameras or scanners integrated into an HMD or other        augmented reality display system or attached or connected to an        HMD or other augmented reality display system and from one or        more cameras or scanners separate from an HMD or other augmented        reality display system (e.g. attached to an OR light, OR        fixture, OR wall, a robot etc.). The tracking data and/or        information can, for example, comprise data of one or more        markers, e.g. attached to one or more HMDs or other augmented        reality display systems, a patient (e.g. an anatomic structure,        landmark or surface of the patient), a physical surgical tool or        instrument, a physical implant or a combination thereof. In some        embodiments, one or more markers can be configured specific for        each HMD or other augmented reality display system, e.g. using a        unique bar code, QR code, fiducial marker configuration, RF        signal etc., which can, for example, be recognized by one or        more computer processors.    -   An instrument calibration module 1110. The one or more computer        processors can be configured to run an instrument calibration        module. The instrument calibration module can comprise an        interface. The instrument calibration module can be configured        for determining an instrument or tool tip, an instrument or tool        axis, an instrument or tool length, an instrument or tool        rotational wobble etc. The tracking module or tracking engine        and/or an interface can be configured for communication with        other operating modules, e.g. integrated with the first        computing system or the second computing system.    -   A headset calibration module 1120. The one or more computer        processors can be configured to run a headset calibration        module. The headset calibration module can comprise an        interface. The headset calibration module, including its        interface, can, for example, be configured for determining an        interpupillary distance, a distance from the eye (e.g. pupil,        iris, retina) to the display (e.g. a waveguide, mirror etc.),        and/or for setting or adjusting an interpupillary distance in        the headset, for setting or adjusting a field of view, for        setting or adjusting a distance from the eye to the display for        a given user, for setting or adjusting user display preferences        (e.g. color, targeting tools, magnification, alphanumeric        display, display window location, alphanumeric data location        etc. The headset calibration module and/or an interface can be        configured for communication with other operating modules, e.g.        integrated with the first computing system or the second        computing system.    -   An imaging and navigation module 1130. The one or more computer        processors can be configured to run an imaging and navigation        module. The imaging and navigation module can, for example,        comprise a PACS interface, an interface to an imaging system,        e.g. x-ray, C-arm, 3D C-arm, cone-beam CT, CT, MRI etc., a DICOM        reader, an image processing and display unit, e.g. for        multi-planar reconstruction and/or display, 3D reconstruction        and/or display, a planning module, e.g. with an interface for a        user, surgeon (e.g. for screw or implant selection and        placement), a navigation module, e.g. with coordinate output        from a planning module. The imaging and navigation module and/or        an interface can be configured for communication with other        operating modules, e.g. integrated with the first computing        system or the second computing system.    -   An AR or VR wireless networking module 1140. In the AR or VR        wireless networking module, one or more computer processors can        be configured for packaging of data, e.g. data listed in        Table 2. Packing of data can, optionally, comprise data        compression. Packaging of data can, optionally, comprise        segmentation or separation into different time segments. Data        packets for different time segments can, for example, be        generated at a rate of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz,        40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz,        100 Hz, or any other rate or frequency including higher rates or        frequencies. One or more computer processors can be configured        to transmit data packets using a wireless access point to a        second, third, fourth or more computing system. One or more        computer processors can be configured to receive data packets        using the wireless access point from a second, third, fourth or        more computing system. In some embodiments, multiple wireless        access points can be used. Any wireless communication protocol        known in the art, e.g. Bluetooth, WiFi, LiFi, can be used.    -   An AR or VR visualization module 1150. One or more computer        processors can be configured for generating display data for        display by one or more HMDs or other augmented reality display        systems. The display data can comprise targeting displays, e.g.        target disk like, magnified displays, color labeled displays        (e.g. red, yellow, green indicating position or alignment        accuracy or thresholds), 3D models of the patient, image slices,        3D models or 3D or 2D graphical representations of tools or        instruments (e.g. tracked physical tools or instruments)        alphanumeric displays, and/or an AR or VR enabled graphical user        interface (e.g. gesture recognition, using gesture recognition,        virtual pointers, a virtual mouse, a virtual keyboard, virtual        buttons, gaze recognition, gaze lock or a combination thereof).        An AR or VR visualization module can comprise a display of a        virtual user interface and/or a display of one or more virtual        interactions, e.g. collisions, with a virtual user interface        (e.g. with a tracked physical tool or instrument). The AR or VR        visualization and/or display module and/or an interface can be        configured for communication with other operating modules, e.g.        integrated with the first computing system or the second        computing system.    -   An AR or VR display module 1160. One or more computer processors        can be configured for generating one or more 3D stereoscopic        views, for example at a rate similar to the data transmission or        reception and/or at a different rate. The 3D stereoscopic view        can, for example, be adjusted for user specific characteristics        (e.g. interpupillary distance, distance from pupil to mirror        etc.). The 3D stereoscopic view can, for example, be adjusted        for the distance from the pupil or the display to the patient,        e.g. a surgical site, for example, in a spine, knee, hip, organ,        vessel etc.; adjustments can comprise adjustments of focal plane        or point, adjustments of scale or magnification of virtual        displays and display items, adjustments of convergence.        Adjustments can be in real-time or near real-time. Adjustments        can be at less than real-time. An AR or VR display module can        comprise a display of a virtual user interface and/or a display        of one or more virtual interactions, e.g. collisions, with a        virtual user interface (e.g. with a tracked physical tool or        instrument).

The second computing system can be part of one or more mobile, wirelessHMD or other augmented reality display system units. A second computingsystem can be part of a first mobile, wireless HMD or other augmentedreality display system unit. A third computing system can be part of asecond mobile wireless HMD or other augmented reality display systemunit. A fourth computing system can be part of a third mobile wirelessHMD or other augmented reality display system unit. A fifth computingsystem can be part of a third mobile wireless HMD or other augmentedreality display system unit, etc. The first, second, third, fourth etc.mobile wireless HMD unit can be worn by a user, e.g. a physician, asurgeon, a dentist, a physician or dental assistant etc. The first,second, third, fourth etc. mobile wireless HMD unit can be a videosee-through HMD. The first, second, third, fourth etc. mobile wirelessHMD unit can be an optical see-through HMD.

One or more modules can be combined or integrated and can, for example,be operated by the same one or more computer processors or, optionally,by different one or more computer processors.

One or more of the computer processors operating a tracking engine ortracking module 1100, one or more computer processors operating aninstrument calibration module 1110, one or more of the computerprocessors operating a headset calibration module 1120, one or more ofthe computer processors operating an imaging and navigation module 1130,one or more of the computer processors operating an AR wirelessnetworking module 1140, one or more of the computer processors operatingan AR visualization module 1150, and/or one or more of the computerprocessors operating an AR display module 1160 can be the same.

One or more of the computer processors operating a tracking engine ortracking module 1100, one or more computer processors operating aninstrument calibration module 1110, one or more of the computerprocessors operating a headset calibration module 1120, one or more ofthe computer processors operating an imaging and navigation module 1130,one or more of the computer processors operating an AR wirelessnetworking module 1140, one or more of the computer processors operatingan AR visualization module 1150, and/or one or more of the computerprocessors operating an AR display module 1160 can be different.

One or more modules can be operated by a first computing system, whileone or more different modules can be operated by a second, or third, orfourth, etc. computing system. One or more modules can be operated by afirst computing system, while one or more different modules can beoperated by a second, and third, and fourth, etc. computing system.

One or more modules can be operated by a first computing system, whileone or more same modules can be operated by a second, or third, orfourth, etc. computing system. One or more modules can be operated by afirst computing system, while one or more same modules can be operatedby a second, and third, and fourth, etc. computing system.

In one example, a first computing system can comprise a tracking moduleor tracking engine 1100, an instrument calibration module 1110, aheadset calibration module 1120, an imaging and navigation module 1130,a AR wireless networking module 1140, and an AR visualization module1150; a second computing system can comprise an AR display module 1160.

In another example, a first computing system can comprise a trackingmodule or tracking engine 1100, an instrument calibration module 1110, aheadset calibration module 1120, an imaging and navigation module 1130,and a AR wireless networking module 1140; a second computing system cancomprise an AR visualization module 1150 and an AR display module 1160.

In another example, a first and a second computing system can compriseone or more of the same modules, for example dedicated to the sameand/or different functions. For example, a first computing system cancomprise an AR wireless networking module 1140, for example for datatransmission; a second computing system can also comprise an AR wirelessnetworking module 1140, for example for data reception.

In another example, a first computing system can comprise a trackingmodule or tracking engine 1100, an instrument calibration module 1110,an imaging and navigation module 1130, and an AR wireless networkingmodule 1140; a second computing system can comprise a headsetcalibration module 1120, an AR wireless networking module 1140, an ARvisualization module 1150 and an AR display module 1160.

Any combination of same and/or different modules, including duplicationof modules on different (first, second, third, fourth, fifth) computingsystems is possible and within the scope of this disclosure.

Using one or more computer processors, e.g. in a second computingsystem, the AR display 1160 module can generate the stereoscopic ornon-stereoscopic view of a first person for the first person'srespective view angle in relationship to one or more anatomic landmarksor anatomic structures of a patient. Using one or more computerprocessors, e.g. in a third computing system, the AR display 1160 modulecan generate the stereoscopic or non-stereoscopic view of a secondperson for the second person's respective view angle in relationship toone or more anatomic landmarks or anatomic structures of the patient.Using one or more computer processors, e.g. in a fourth computingsystem, the AR display 1160 module can generate the stereoscopic ornon-stereoscopic view of a third person for the third person'srespective view angle in relationship to one or more anatomic landmarksor anatomic structures of the patient. Using one or more computerprocessors, e.g. in a fifth computing system, the AR display 1160 modulecan generate the stereoscopic or non-stereoscopic view of a fourthperson for the fourth's person's respective view angle in relationshipto one or more anatomic landmarks or anatomic structures of the patient,etc.

The second, third, fourth, fifth or more computing systems can be thesame. The second, third, fourth, fifth or more computing systems can bedifferent, e.g. integrated or connected to different mobile units and/ordifferent HMDs or other augmented reality display systems.

A first, second, third, fourth, fifth or more computing systems can bethe same. A first, second, third, fourth, fifth or more computingsystems can be different. A first, second, third, fourth, fifth or morecomputer processors can be the same. A first, second, third, fourth,fifth or more computer processors can be different.

A first, second, third, fourth, fifth or more computer processor canhave the same processing speed. At least one of a first, second, third,fourth, fifth or more or more computer processor can have a differentprocessing speed. For example, a computer processor can have aprocessing speed of 1 GHz, 1.5 GHz, 2.0 GHz, 2.1 GHz, 2.2 GHz, 2.3 GHz,2.4 GHz, 2.5 GHz, 2.6 GHz, 2.7 GHz, 2.8 GHz, 2.9 Ghz, 3.0 GHz orgreater. Any value is possible. Some applications of the disclosure canbenefit from higher processing speeds, e.g. above 1.5 GHz or 2.0 GHz,for example when data intense, complex data packets are being acquired,generated, transmitted and/or received (see Table 2 also). A computerprocessor can, for example, be a Qualcomm Snapdragon 845 or later(Qualcomm, San Diego, Calif. 92121).

Unicast, Multicast, or Broadcast Transmission and/or Reception

Unicast

In some embodiments, data or data packets, e.g. as listed in Table 2,can be transmitted and/or received with unicast transmission and/orreception, for example between a first computing system or server and asecond computing system or client; the second computing system can beconfigured to generate the stereoscopic or non-stereoscopic 2D or 3Ddisplay 1160 by the HMD or other augmented reality display system.

In some embodiments, a first unicast transmission can be transmittedfrom a first computing system and received by a second computing system,e.g. integrated into or connected to a first HMD or other augmentedreality display system, with the first unicast transmission comprisingthe specific tracking information for the first HMD or other augmentedreality display system and, optionally, instrument and/or surgical sitetracking data. A second, third, fourth, fifth or more unicasttransmission can be transmitted from a first computing system to athird, fourth, fifth, sixth or more computing system, e.g. integratedinto or connected to a second, third, fourth, fifth or more HMD or otheraugmented reality display system, respectively. The second, third,fourth, fifth or more unicast transmission can be sequential, e.g.overlapping or non-overlapping.

A second unicast transmission can be transmitted from a first computingsystem and received by a third computing system, e.g. integrated into orconnected to a second HMD or other augmented reality display system,with the second unicast transmission comprising the specific trackinginformation for the second HMD or other augmented reality display systemand, optionally, instrument and/or surgical site tracking data.

A third unicast transmission can be transmitted from a first computingsystem and received by a fourth computing system, e.g. integrated intoor connected to a third HMD or other augmented reality display system,with the third unicast transmission comprising the specific trackinginformation for the third HMD or other augmented reality display systemand, optionally, instrument and/or surgical site tracking data.

A fourth unicast transmission can be transmitted from a first computingsystem and received by a fifth computing system, e.g. integrated into orconnected to a fourth HMD or other augmented reality display system,with the fourth unicast transmission comprising the specific trackinginformation for the fourth HMD or other augmented reality display systemand, optionally, instrument and/or surgical site tracking data.

A fifth unicast transmission can be transmitted from a first computingsystem and received by a sixth computing system, e.g. integrated into orconnected to a fifth HMD or other augmented reality display system, withthe fifth unicast transmission comprising the specific trackinginformation for the fifth HMD or other augmented reality display systemand, optionally, instrument and/or surgical site tracking data.

Any number of unicast transmissions can be transmitted from a firstcomputing system and received by a corresponding number of HMDs or otheraugmented reality display systems. If an overall transmission andreception rate of 30 Hz, 40 Hz, 50 Hz, 60 Hz, or 70 Hz is desired, thesequential unicast transmissions can be completed within 0.0333 sec,0.025 sec, 0.02 sec, 0.0166 sec, 0.01428 sec, or any other value. Thenext round of sequential unicast transmissions and/or receptions canthen start in order to achieve near real-time or real-time transmissionand/or reception of specific tracking information for the differentheadsets, along with, optionally, instrument tracking and/or patienttracking data for stereoscopic and/or non-stereoscopic display ofinstrument and/or patient data by the one or more HMDs or otheraugmented reality display systems.

Multicast, Broadcast

In some embodiments, data or data packets, e.g. as listed in Table 2,can be transmitted and/or received with multicast transmission and/orreception, for example between a first computing system or server andmultiple clients, e.g. a second computing system, third computingsystem, fourth computing system, fifth computing system, etc.,optionally each with one or more computer processors; the secondcomputing system, third computing system, fourth computing system, fifthcomputing system, etc. can be configured to generate the stereoscopic ornon-stereoscopic 2D or 3D display 1160 by the corresponding first,second, third, and fourth etc. HMDs or other augmented reality displaysystems.

In some embodiments, data or data packets, e.g. as listed in Table 2,can be transmitted and/or received with broadcast transmission and/orreception, for example between a first computing system or server andmultiple clients (for example all available clients), e.g. a secondcomputing system, third computing system, fourth computing system, fifthcomputing system, etc., optionally each with one or more computerprocessors; the second computing system, third computing system, fourthcomputing system, fifth computing system, etc. can be configured togenerate the stereoscopic or non-stereoscopic 2D or 3D display 1160 bythe corresponding first, second, third, and fourth etc. HMDs or otheraugmented reality display systems.

With multicast or broadcast transmission and/or reception, the position,orientation, position and orientation, direction of movement and/ortracking data for each HMD or other augmented reality display system canbe labelled for each HMD or other augmented reality display system, e.g.corresponding to the HMD or other augmented reality display systemnumber, for example label “1” for the first HMD or other augmentedreality display system, label “2” for the second HMD or other augmentedreality display system, label “3” for the third HMD or other augmentedreality display system, label “4” for the fourth HMD or other augmentedreality display system, label “5” for the fifth HMD or other augmentedreality display system, etc. The second computing system, thirdcomputing system, fourth computing system, fifth computing system, etc.can be configured to generate the stereoscopic or non-stereoscopic 2D or3D display 1160 for the corresponding first, second, third, and fourthetc. HMDs or other augmented reality display systems based on the labelscorresponding to each HMD or other augmented reality display system andtracking data for each respective HMD or other augmented reality displaysystem. For example, the second computing system can identify the labelfor the first HMD or other augmented reality display system, e.g. “1”,in the received data and generate the stereoscopic or non-stereoscopic2D or 3D display for the first HMD or other augmented reality displaysystem using the HMD or other augmented reality display system trackingdata labeled for the first HMD or other augmented reality displaysystem; the third computing system can identify the label for the secondHMD or other augmented reality display system, e.g. “2”, in the receiveddata and generate the stereoscopic or non-stereoscopic 2D or 3D displayfor the second HMD or other augmented reality display system using theHMD or other augmented reality display system tracking data labeled forthe second HMD or other augmented reality display system; the fourthcomputing system can identify the label for the third HMD or otheraugmented reality display system, e.g. “3”, in the received data andgenerate the stereoscopic or non-stereoscopic 2D or 3D display for thethird HMD or other augmented reality display system using the HMD orother augmented reality display system tracking data labeled for thethird HMD or other augmented reality display system; the fifth computingsystem can identify the label for the fourth HMD or other augmentedreality display system, e.g. “4”, in the received data and generate thestereoscopic or non-stereoscopic 2D or 3D display for the fourth HMD orother augmented reality display system using the HMD or other augmentedreality display system tracking data labeled for the fourth HMD or otheraugmented reality display system; and so forth for any number of HMDs orother augmented reality display systems used. In this manner, eachclient or second, third, fourth, fifth etc. computing system, optionallywith one or more computer processors, can generate the stereoscopic ornon-stereoscopic 2D or 3D display or augmented view for each HMD orother augmented reality display system with the correct view angle andviewing perspective for each specific HMD or other augmented realitydisplay system, for example in relationship to one or more trackedanatomic structures of the patient and/or one or more tracked physicaltools, instruments and/or implants and/or one or more markers attachedto a patient, e.g. a fiducial array attached to a bone. For example,each client or second, third, fourth, fifth etc. computing system,optionally with one or more computer processors, can generate thestereoscopic or non-stereoscopic 2D or 3D display for each HMD or otheraugmented reality display system with the correct view angle and viewingperspective for each specific HMD or other augmented reality displaysystem for a virtual display, e.g. a virtual user interface or displayof one or more interactions of a tracked physical surgical tool orinstrument with a virtual user interface, for example in relationship toone or more tracked anatomic structures of the patient and/or one ormore tracked physical tools, instruments and/or implants and/or one ormore markers attached to a patient, e.g. a fiducial array attached to abone. In this manner, a display of a virtual user interface or of one ormore interactions therewith can, for example, be displayed in thedisplay plane of the physical HMD unit, e.g. a waveguide display ormirror based display. In this manner, for example, a display of avirtual user interface or of one or more interactions therewith can bedisplayed in a predetermined display plane for a first, second, third,fourth, fifth and/or sixth HMDs, for example a display planesubstantially parallel to the user's retina or a display planesubstantially perpendicular to one or more pupillary axes of the user'seyes. In other embodiments, a display of a virtual user interface or ofone or more interactions therewith can be displayed in a predeterminedposition and/or orientation for a first, second, third, fourth, fifthand/or sixth HMDs, for example a display plane at a predeterminedposition and/or orientation in relationship to a patient, a surgicalsite, one or more markers attached to the patient, e.g. a fiducial arrayattached to a bone, one or more markers attached to a structure in anoperating room, e.g. an OR table, OR light etc.

Network of HMD or Other Augmented Reality Display System Systems

In some embodiments, a network of HMDs or other augmented realitydisplay systems can be used. One or more HMDs or other augmented realitydisplay systems can comprise at least one camera, scanner, 3D scanner,LIDAR system, depth sensor, IMU or a combination thereof integrated intoor attached to the HMD or other augmented reality display system. The atleast one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMUor a combination thereof integrated into or attached to the one or moreHMDs or other augmented reality display systems can be used to generatecoordinate and/or tracking information of one or more HMDs or otheraugmented reality display systems, a patient, an anatomic structure of apatient, one or more physical surgical tools, one or more physicalsurgical instruments, one or more robot (e.g. a robot with a roboticarm, a handheld robot, or a combination thereof) or a combinationthereof.

Two or more of the HMDs or other augmented reality display systems canoptionally interconnect and create a network, e.g. for a sharedexperience of the augmented views and/or for multi-directionalgeneration of coordinate information and/or tracking information. Theuse of multi-directional generation of coordinate information and/ortracking information can be helpful to reduce or avoid line of sightissues. For example, when the line of sight is blocked for a firstcamera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or acombination thereof integrated into or attached to a first HMD, the lineof sight can be intact or maintained for a second, third, fourth, fifth,etc. or combination thereof camera, scanner, 3D scanner, LIDAR system,depth sensor, IMU or a combination thereof integrated into or attachedto a first, second, third, fourth of fifth, etc. HMD.

The HMDs or other augmented reality display systems can be organized ina client-server network where multiple HMD clients can centralizedaround a single server, e.g. a first computing unit. Thus, HMD devicescan be relieved of computing power when outsourcing tasks which arecomputational intensive (image processing) to the server. Moreover,battery life of the HMD's can be significantly prolonged which can makethe approach attractive even in case of a single HMD client. The servercan be accessible in the OR.

In case of multiple clients, different data inputs from the variousperspectives (e.g. from a first, second, third, fourth, fifth etc. HMD)can be used by the server to increase the accuracy of the calculations(e.g. by averaging out errors). In some embodiments, coordinateinformation and/or tracking information, e.g. from spatial maps from twoor more HMD clients, can be obtained and processed by the server. Forexample, spatial maps can consist of triangular meshes built from eachHMD's depth sensor information. Once spatial maps have been transferredfrom a first, second, third, fourth, fifth or combination there of HMDto the server, the different meshes can be combined into a combined,more accurate mesh using, for example, an averaging algorithm: Forexample, the data from a first HMD can be used as the baseline. Fromeach face in the baseline mesh, a ray can be cast along the surfacenormal of the face. Intersection points between the ray and all othermeshes can be calculated. A new vertex for the combined mesh can bederived as the average of all intersection points along the ray. The newvertices from adjacent triangles in the baseline mesh can be connectedto form the faces in the combined mesh. The combined mesh can then betransferred back to the individual HMD's for refinement of theregistration, coordinate or tracking information and/or for refinementof the real-time or near real-time updating of the stereoscopic ornon-stereoscopic HMD display, e.g. superimposed and/or aligned with ananatomic structure or anatomic landmark of a patient.

In some embodiments, once coordinate information, registrationinformation, tracking information, surface information, e.g. of one ormore HMDs or other augmented reality display systems, a patient, ananatomic structure of a patient, one or more physical surgical tools,one or more physical surgical instruments, one or more robot or acombination thereof has been obtained using at least one camera,scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combinationthereof integrated into or attached to the HMDs or other augmentedreality display systems and the information has been transferred from afirst, second, third, fourth, fifth or combination there of HMD to theserver, the data generated by the at least one camera, scanner, 3Dscanner, LIDAR system, depth sensor, IMU or a combination thereofintegrated into or attached to the two or more HMDs or other augmentedreality display systems can be combined, e.g. into a combined, moreaccurate surface or surface mesh using, for example, an averagingalgorithm. Optionally, a weighting can be applied to the datatransferred by different HMDs or other augmented reality displaysystems, e.g. with a higher weight for HMDs or other augmented realitydisplay systems located closer to the patient and/or closer to the atleast one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMUor a combination thereof.

Intrinsic and/or Extrinsic Tracking of Surgical Robots

In some embodiments, surgical robots can comprise a robotic arm, ahandheld robot, handheld portions, or a combination thereof. In someembodiments, surgical robots can comprise one or more sensor, camera,video system, scanner, e.g. 3D scanner, LIDAR system, depth sensor,controller, electric controller, mechanical controller, drive, actuator,end effector, attachment mechanism, potentiometer, inertial measurementunit, accelerometer, magnetometer, gyroscope, force sensor, pressuresensor, position sensor, orientation sensor, motion sensor, wire, stepmotor, electric motors, hydraulic motor, electric and/or mechanicalactuator, switch, display unit, computer processor, or a combinationthereof.

In some embodiments, coordinate information, tracking data or acombination thereof of one or more end effectors, physical tools orinstruments integrated or attached to a or part of a robot and/or of oneor more physical implants, physical implant components, or physicaltrial implants attached to a robot can be generated with use of posedata, sensor data, camera data, 3D scanner data, controller data, drivedata, actuator data, end effector data or a combination thereof of therobot or one or more robot components, for example obtained usingintrinsic or internal data generated by or including intrinsic orinternal, integrated or attached sensors, potentiometers, cameras, videosystems, 3D scanners, LIDAR systems, depth sensors, inertial measurementunits, accelerometers, magnetometers, gyroscopes, force sensors,pressure sensors, position sensors, orientation sensors, motion sensors,position and/or orientation feedback from robot step motors, positionand/or orientation feedback from robot electric motors, position and/ororientation feedback from robot hydraulic motors, position and/ororientation feedback from robot electric and/or mechanical actuators,position and/or orientation feedback from robot drives, position and/ororientation feedback from robotic controllers, position and/ororientation feedback from one or more robotic computer processors, or acombination thereof. If one or more integrated or attached cameras,video systems, 3D scanners, LIDAR systems, depth sensors, or acombination thereof is used for generating intrinsic or internal robotdata, the data can optionally be corrected for any distance and/orangular offset between the one or more integrated or attached cameras,video systems, 3D scanners, LIDAR systems, depth sensors, or acombination thereof and an end effector, a surgical tool or instrumentattached to or integrated into or part of the robot, e.g. a cuttingtool, tissue removal tool (e.g. a drill, saw, reamer, impactor), or anablation tool. Alternatively and/or additionally, If one or moreintegrated or attached cameras, video systems, 3D scanners, LIDARsystems, depth sensors, or a combination thereof is used for generatingintrinsic or internal robot data, the data can optionally be correctedfor any distance and/or angular offset between the one or moreintegrated or attached cameras, video systems, 3D scanners, LIDARsystems, depth sensors, or a combination thereof and an anatomicstructure, surface and/or landmark of a patient. Any combination ofoffset, e.g. distance and/or angle, correction is possible. In someembodiments, one or more cameras, video systems, 3D scanners, LIDARsystems, depth sensors external to a robot (e.g. on a stand, in an ORlight and/or one or more HMDs or other augmented reality displaysystems) can be used for determining the distance and/or angle offset.

In some embodiments, coordinate information, tracking data or acombination thereof of one or more end effectors, physical tools orinstruments integrated or attached to or part of a robot and/or of oneor more physical implants, physical implant components, or physicaltrial implants attached to a robot can be obtained or generated with useof one or more cameras, video systems, 3D scanners, LIDAR systems, depthsensors, or combination thereof extrinsic or external to the robot and,for example, integrated or attached to one or more HMDs or otheraugmented reality display systems, separate from one or more HMDs orother augmented reality display systems (e.g. on a stand, tripod,attached to or integrated into OR lighting, OR fixtures, an imagingsystem (e.g. x-ray, cone beam CT, CT)), or a combination thereof. One ormore computer processors can be configured, for example, using the oneor more cameras, video systems, 3D scanners, LIDAR systems, depthsensors, or combination thereof extrinsic or external to the robot, todetermine the position, orientation, direction of movement, one or morecoordinates, or combination thereof of at least a portion of the one ormore end effectors, physical surgical tools, at least a portion of therobot, or a combination thereof (e.g. using image processing and/orpattern recognition and/or an artificial neural network) or of one ormore markers, e.g. active markers (e.g. RF markers), passive markers(e.g. infrared markers), optical markers (e.g. with geometric patterns,QR codes, bar codes, defined shapes, e.g. triangles, squares, rectanglesetc.), LEDs or a combination thereof integrated or attached to the oneor more end effectors, physical tools or instruments, integrated orattached to at least portions of the robot, or a combination thereof(extrinsic or external data).

In some embodiments, one or more displays by one or more computermonitors or by one or more HMDs or other augmented reality displaysystems can be generated, wherein the display can be non-stereoscopic(e.g. by the computer monitor, other augmented reality display device(s)and/or the HMD) or stereoscopic (e.g. by the HMD).

In some embodiments, one or more computer processors can generate adisplay, e.g. by a computer monitor and/or one or more HMDs or otheraugmented reality display systems, of virtual data, e.g. a virtualsurgical plan, one or more virtual surgical guides (e.g. a virtual axis,virtual plane, virtual gut guide) and/or one or more patient surface(s)using intrinsic or internal robot data, e.g. registration data,coordinate data, and/or tracking data of one or more HMDs or otheraugmented reality display systems, a patient, an anatomic structure of apatient, one or more physical surgical tools, one or more physicalsurgical instruments, one or more robot, or a combination thereof.

In some embodiments, one or more computer processors can generate adisplay, e.g. by a computer monitor and/or one or more HMDs or otheraugmented reality display systems, of virtual data, e.g. a virtualsurgical plan, one or more virtual surgical guides (e.g. a virtual axis,virtual plane, virtual gut guide) and/or one or more patient surface(s)using extrinsic or external robot data, e.g. registration data,coordinate data, and/or tracking data of end effectors, one or morephysical surgical tools, one or more physical surgical instruments, oneor more robot, or a combination thereof.

In some embodiments, one or more computer processors can generate adisplay, e.g. by a computer monitor and/or one or more HMDs or otheraugmented reality display systems, of virtual data, e.g. a virtualsurgical plan, one or more virtual surgical guides (e.g. a virtual axis,virtual plane, virtual gut guide) and/or one or more patient surface(s)using intrinsic or internal and extrinsic or external robot data, e.g.registration data, coordinate data, and/or tracking data of one or moreHMDs or other augmented reality display systems, a patient, an anatomicstructure of a patient, one or more physical surgical tools, one or morephysical surgical instruments, one or more robot, or a combinationthereof. In this example, intrinsic and or internal robot data can,optionally, be displayed using a different color or display pattern thanextrinsic or external robot data, thereby highlighting potentialdifferences and/or deviations. In some embodiments, one or more computerprocessors can be used to compute any differences and/or deviationsbetween intrinsic or internal and extrinsic or external robot data, e.g.a difference in a projected instrument or tool path, e.g. a drill path,a saw path, a difference in a planned or executed tissue resection. Oneor more computer processors can be configured to generate a differencedisplay, for example by a computer monitor and/or one or more HMDs orother augmented reality display systems, e.g. using color coding, lineor bar charts or any other chart known in the art, and/or alphanumericdisplay. The difference between intrinsic or internal and extrinsic orexternal robot data, e.g. registration data, coordinate data, and/ortracking data of one or more HMDs or other augmented reality displaysystems, a patient, an anatomic structure of a patient, one or morephysical surgical tools, one or more physical surgical instruments, oneor more robot, or a combination thereof can be used to highlight anypotential deviation of a robot from a predetermined plan, e.g. apredetermined tissue resection (for example a predetermined tissueresection volume, tissue resection area, tissue resection surface, bonecut, drilling, reaming, milling, impacting).

Intrinsic and/or Extrinsic Tracking of Imaging Systems and ImagingSystem Components

In some embodiments, an imaging system can comprise one or more imagingsystem components. In some embodiments, one or more imaging systemcomponents can comprise one or more sensor, camera, video system,scanner, e.g. 3D scanner, LIDAR system, depth sensor, controller,electric controller, mechanical controller, drive, actuator, endeffector, attachment mechanism, potentiometer, inertial measurementunit, accelerometer, magnetometer, gyroscope, force sensor, pressuresensor, position sensor, orientation sensor, motion sensor, wire, stepmotor, electric motors, hydraulic motor, electric and/or mechanicalactuator, switch, display unit, computer processor, or a combinationthereof.

In some embodiments, coordinate information, tracking data or acombination thereof of one or more imaging system components can begenerated with use of pose data, sensor data, camera data, 3D scannerdata, controller data, drive data, actuator data, end effector data or acombination thereof of the one or more imaging system components, forexample obtained using intrinsic or internal data generated by orincluding intrinsic or internal, integrated or attached sensors,potentiometers, cameras, video systems, 3D scanners, LIDAR systems,depth sensors, inertial measurement units, accelerometers,magnetometers, gyroscopes, force sensors, pressure sensors, positionsensors, orientation sensors, motion sensors, position and/ororientation feedback from imaging system component step motors, positionand/or orientation feedback from imaging system component electricmotors, position and/or orientation feedback from imaging systemcomponent hydraulic motors, position and/or orientation feedback fromsystem component electric and/or mechanical actuators, position and/ororientation feedback from imaging system component drives, positionand/or orientation feedback from imaging system component controllers,position and/or orientation feedback from imaging system componentcomputer processors, or a combination thereof. If one or more integratedor attached cameras, video systems, 3D scanners, LIDAR systems, depthsensors, or a combination thereof is used for generating intrinsic orinternal imaging system component data, the data can optionally becorrected for any distance and/or angular offset between the one or moreintegrated or attached cameras, video systems, 3D scanners, LIDARsystems, depth sensors, or a combination thereof and one or more imagingsystem components. Alternatively and/or additionally, If one or moreintegrated or attached cameras, video systems, 3D scanners, LIDARsystems, depth sensors, or a combination thereof is used for generatingintrinsic or internal imaging system component data, the data canoptionally be corrected for any distance and/or angular offset betweenthe one or more integrated or attached cameras, video systems, 3Dscanners, LIDAR systems, depth sensors, or a combination thereof and ananatomic structure, surface and/or landmark of a patient. Anycombination of offset, e.g. distance and/or angle, correction ispossible. In some embodiments, one or more cameras, video systems, 3Dscanners, LIDAR systems, depth sensors external to one or more imagingsystem components (e.g. on a stand, in an OR light and/or one or moreHMDs or other augmented reality display systems) can be used fordetermining the distance and/or angle offset.

In some embodiments, coordinate information, tracking data or acombination thereof of one or more imaging system components can beobtained or generated with use of one or more cameras, video systems, 3Dscanners, LIDAR systems, depth sensors, or combination thereof extrinsicor external to the imaging system components and, for example,integrated or attached to one or more HMDs or other augmented realitydisplay systems, separate from one or more HMDs or other augmentedreality display systems (e.g. on a stand, tripod, attached to orintegrated into OR lighting, OR fixtures, an imaging system (e.g. x-ray,cone beam CT, CT)), or a combination thereof. One or more computerprocessors can be configured, for example, using the one or morecameras, video systems, 3D scanners, LIDAR systems, depth sensors, orcombination thereof extrinsic or external to the one or more imagingsystem components, to determine the position, orientation, direction ofmovement, one or more coordinates, or combination thereof of at least aportion of the one or more imaging system components (e.g. using imageprocessing and/or pattern recognition and/or an artificial neuralnetwork) or of one or more markers, e.g. active markers (e.g. RFmarkers), passive markers (e.g. infrared markers), optical markers (e.g.with geometric patterns, QR codes, bar codes, defined shapes, e.g.triangles, squares, rectangles etc.), LEDs or a combination thereofintegrated or attached to the one or more imaging system components(extrinsic or external data).

In some embodiments, one or more displays by one or more computermonitors or by one or more HMDs or other augmented reality displaysystems can be generated, wherein the display can be non-stereoscopic(e.g. by the computer monitor, other augmented reality display device(s)and/or the HMD) or stereoscopic (e.g. by the HMD).

In some embodiments, one or more computer processors can generate adisplay, e.g. by a computer monitor and/or one or more HMDs or otheraugmented reality display systems, of virtual data, e.g. a virtualsurgical plan, one or more virtual surgical guides (e.g. a virtual axis,virtual plane, virtual gut guide), one or more pre-operative orintra-operative imaging data and/or one or more patient surface(s) usingintrinsic or internal imaging system data, e.g. registration data,coordinate data, and/or tracking data of one or more imaging systemcomponents.

In some embodiments, one or more computer processors can generate adisplay, e.g. by a computer monitor and/or one or more HMDs or otheraugmented reality display systems, of virtual data, e.g. a virtualsurgical plan, one or more virtual surgical guides (e.g. a virtual axis,virtual plane, virtual gut guide) one or more pre-operative orintra-operative imaging data and/or one or more patient surface(s) usingextrinsic or external tracking data, e.g. registration data, coordinatedata, and/or tracking data of one or more HMDs or other augmentedreality display systems, a patient, an anatomic structure of a patient,one or more physical surgical tools, one or more physical surgicalinstruments, one or more imaging system components, or a combinationthereof.

In some embodiments, one or more computer processors can generate adisplay, e.g. by a computer monitor and/or one or more HMDs or otheraugmented reality display systems, of virtual data, e.g. a virtualsurgical plan, one or more virtual surgical guides (e.g. a virtual axis,virtual plane, virtual gut guide) and/or one or more patient surface(s)using intrinsic or internal and extrinsic or external data, e.g.registration data, coordinate data, and/or tracking data of one or moreHMDs or other augmented reality display systems, a patient, an anatomicstructure of a patient, one or more physical surgical tools, one or morephysical surgical instruments, one or more imaging system components, ora combination thereof. In this example, intrinsic and or internalimaging system data can, optionally, be displayed using a differentcolor or display pattern than extrinsic or external imaging system data,thereby highlighting potential differences and/or deviations. In someembodiments, one or more computer processors can be used to compute anydifferences and/or deviations between intrinsic or internal andextrinsic or external imaging system data. One or more computerprocessors can be configured to generate a difference display, forexample by a computer monitor and/or one or more HMDs or other augmentedreality display systems, e.g. using color coding, line or bar charts orany other chart known in the art, and/or alphanumeric display.

Aspects of the disclosure relate to a system comprising at least onehead mounted display or other augmented reality display device, at leastone camera or scanning device, a first computing system comprising oneor more computer processors and a second computing system comprising oneor more computer processors,

wherein the first computing system is configured to obtain real-timetracking information of the at least one head mounted display or otheraugmented reality display device, of at least one anatomic structure ofa patient, and of at least one physical surgical tool or physicalsurgical instrument using the at least one camera or scanning device,

wherein the first computing system is configured for wirelesstransmission of the real-time tracking information of the at least onehead mounted display or other augmented reality display device, the atleast one anatomic structure of the patient, and the at least onephysical surgical tool or physical surgical instrument,

wherein the second computing system is connected to or integrated intothe at least one head mounted display or other augmented reality displaydevice,

wherein the second computing system is configured for wireless receptionof the real-time tracking information of the at least one head mounteddisplay or other augmented reality display device, the at least oneanatomic structure of the patient, and the at least one physicalsurgical tool or physical surgical instrument, and

wherein the second computing system is configured to generate a 3Dstereoscopic view, wherein the stereoscopic view comprises a 3Drepresentation of the at least one physical surgical tool or physicalsurgical instrument.

In some embodiments, the one or more computer processors of the secondcomputing system generate the 3D stereoscopic view for the view angle ofthe head mounted display or other augmented reality display devicerelative to the at least one anatomic structure of the patient using thereal-time tracking information of the at least one head mounted displayor other augmented reality display device.

In some embodiments, the real-time tracking information comprisestracking information of two or more head mounted display or otheraugmented reality display devices.

In some embodiments, the real-time tracking information comprises a headmounted display or other augmented reality display device specific labelfor each head mounted display or other augmented reality display device.In some embodiments, the real-time tracking information is labeled foreach tracked head mounted display or other augmented reality displaydevice.

In some embodiments, the real-time tracking information comprisestracking information of two or more head mounted display or otheraugmented reality display devices. In some embodiments, the two or morehead mounted display or other augmented reality display devices arelocated in different locations. In some embodiments, the real-timetracking information comprises a head mounted display or other augmentedreality display device specific label for each head mounted display orother augmented reality display device. In some embodiments, thereal-time tracking information is labeled for each tracked head mounteddisplay or other augmented reality display device.

In some embodiments, the one or more computer processors of the secondcomputing system generate the 3D stereoscopic view for an interpupillarydistance adjusted for a person wearing the head mounted display or otheraugmented reality display device.

In some embodiments, the second computing system is integrated with theat least one head mounted display or other augmented reality displaydevice.

In some embodiments, the second computing system is separate from the atleast one head mounted display or other augmented reality display deviceand is connected to the display unit of the at least one head mounteddisplay or other augmented reality display device using at least onecable.

In some embodiments, the wireless transmission or reception ortransmission and reception comprises a WiFi signal, a LiFi signal, aBluetooth signal or a combination thereof.

In some embodiments, the camera or scanning device is separate from theat least one head mounted display or other augmented reality displaydevice.

In some embodiments, the camera or scanning device is integrated orattached to the at least one head mounted display or other augmentedreality display device.

In some embodiments, the wireless transmission comprises sending datapackets comprising the real-time tracking information of the at leastone head mounted display or other augmented reality display device, theat least one anatomic structure of a patient, and the at least onephysical surgical tool or physical surgical instrument, at a rate of 20Hz or greater.

In some embodiments, the wireless reception comprises receiving datapackets comprising the real-time tracking information of the at leastone head mounted display or other augmented reality display device, theat least one anatomic structure of a patient, and the at least onephysical surgical tool or physical surgical instrument, at a rate of 20Hz or greater.

In some embodiments, the system comprising a third computing system,wherein the third computing system is configured for wireless receptionof the real-time tracking information from the first computing systemand wherein the third computing system is configured for wirelesstransmission of the real-time tracking information to the secondcomputing system.

In some embodiments, the third computing system comprises a chain ofcomputing systems configured for wireless reception and wirelesstransmission of the real-time tracking information.

In some embodiments, the system comprises a third computing system,wherein the third computing system is connected to or integrated into asecond head mounted display or other augmented reality display device,wherein the third computing system is configured for wireless receptionof the real-time tracking information of the second head mounted displayor other augmented reality display device, the at least one anatomicstructure of a patient, and the at least one physical surgical tool orphysical surgical instrument, wherein the third computing system isconfigured to generate a 3D stereoscopic view by the second head mounteddisplay or other augmented reality display device using the trackinginformation of the second head mounted display or other augmentedreality display device.

In some embodiments, the tracking information of the second head mountedcomprises a label specific to the second head mounted display or otheraugmented reality display device for identifying the trackinginformation of the second head mounted display or other augmentedreality display device by the third computing system.

In some embodiments, the system comprising a fourth computing system,wherein the fourth computing system is connected to or integrated into athird head mounted display or other augmented reality display device,wherein the fourth computing system is configured for wireless receptionof the real-time tracking information of the third head mounted displayor other augmented reality display device, the at least one anatomicstructure of a patient, and the at least one physical surgical tool orphysical surgical instrument, and wherein the fourth computing system isconfigured to generate a 3D stereoscopic view by the third head mounteddisplay or other augmented reality display device using the trackinginformation of the third head mounted display or other augmented realitydisplay device.

In some embodiments, the tracking information of the third head mountedcomprises a label specific to the third head mounted display or otheraugmented reality display device for identifying the trackinginformation of the third head mounted display or other augmented realitydisplay device by the fourth computing system.

In some embodiments, the system comprises a fifth computing system,wherein the fifth computing system is connected to or integrated into afourth head mounted display or other augmented reality display device,wherein the fifth computing system is configured for wireless receptionof the real-time tracking information of the fourth head mounted displayor other augmented reality display device, the at least one anatomicstructure of a patient, and the at least one physical surgical tool orphysical surgical instrument, and wherein the fifth computing system isconfigured to generate a 3D stereoscopic view by the fourth head mounteddisplay or other augmented reality display device using the trackinginformation of the fourth head mounted display or other augmentedreality display device.

In some embodiments, the tracking information of the fourth head mountedcomprises a label specific to the fourth head mounted display or otheraugmented reality display device for identifying the trackinginformation of the fourth head mounted display or other augmentedreality display device by the fifth computing system.

In some embodiments, the real-time tracking information comprises one ormore coordinates.

In some embodiments, the one or more coordinates comprise coordinates ofthe at least one anatomic structure of the patient.

In some embodiments, the one or more coordinates comprise coordinates ofthe at least one physical surgical tool or physical surgical instrument.

In some embodiments, the one or more coordinates comprise coordinates ofthe at least one head mounted display or other augmented reality displaydevice.

In some embodiments, the at least one head mounted display or otheraugmented reality display device comprises at least one opticalsee-through head mounted display or other augmented reality displaydevice.

In some embodiments, the at least one head mounted display or otheraugmented reality display device comprises at least one videosee-through head mounted display or other augmented reality displaydevice.

In some embodiments, the at least one camera, the at least one scanningdevice or the at least one camera and the at least one scanning devicecomprises a laser scanner, a time-of-flight 3D laser scanner, astructured-light 3D scanner, a hand-held laser scanner, a LIDAR scanner,a time-of-flight camera, a depth camera, a video system, a stereoscopiccamera system, a camera array, or a combination thereof.

In some embodiments, the system comprises at least one inertialmeasurement unit. In some embodiments, the at least one inertialmeasurement unit is integrated or attached to the at least one physicalsurgical tool or physical surgical instrument. In some embodiments, theat least one inertial measurement unit is integrated or attached to theat least one anatomic structure of the patient. In some embodiments, theat least one inertial measurement unit is integrated or attached to theat least one head mounted display or other augmented reality displaydevice.

In some embodiments, the real-time tracking information of the at leastone head mounted display or other augmented reality display devicecomprises information from the at least one inertial measurement unit.

Aspects of the disclosure relate to a system comprising two or more headmounted display or other augmented reality display devices, at least onecamera or scanning device, a first computing system comprising one ormore computer processors,

wherein the first computing system is configured to obtain real-timetracking information of at least one anatomic structure of a patient, ofat least one physical surgical tool or physical surgical instrument, andof the two or more head mounted display or other augmented realitydisplay devices, using the at least one camera or scanning device,

wherein the tracking information of the two or more head mounted displayor other augmented reality display devices is labeled specific for eachhead mounted display or other augmented reality display device,

wherein the first computing system is configured for wirelesstransmission of the real-time tracking information of the at least oneanatomic structure of the patient, the tracking information of the atleast one physical surgical tool or physical surgical instrument, andthe labeled tracking information of the two or more head mounted displayor other augmented reality display devices,

a second computing system,

wherein the second computing system is connected to or integrated into afirst of the two or more head mounted display or other augmented realitydisplay devices,

wherein the second computing system is configured for wireless receptionof the real-time tracking information of the at least one anatomicstructure of the patient, the tracking information of the at least onephysical surgical tool or physical surgical instrument, and the labeledtracking information of the first of the two or more head mounteddisplay or other augmented reality display devices,

wherein the second computing system is configured to generate a 3Dstereoscopic display specific for the viewing perspective of the firsthead mounted display or other augmented reality display device using thelabeled tracking information of the first head mounted display or otheraugmented reality display device,

a third computing system,

wherein the third computing system is connected to or integrated into asecond of the two or more head mounted display or other augmentedreality display devices,

wherein the third computing system is configured for wireless receptionof the real-time tracking information of the at least one anatomicstructure of the patient, the tracking information of the at least onephysical surgical tool or physical surgical instrument, and the labeledtracking information of the second of the two or more head mounteddisplay or other augmented reality display devices,

wherein the third computing system is configured to generate a 3Dstereoscopic display specific for the viewing perspective of the secondhead mounted display or other augmented reality display device using thelabeled tracking information of the second head mounted display or otheraugmented reality display device,

wherein the stereoscopic view comprises a 3D representation of the atleast one physical surgical tool or physical surgical instrument.

Virtual User Interface

In some embodiments, a physical tool or instrument (see Table 2), e.g. atracked pointer, a tracked stylus, a tracked tool, a tracked instrumentor a combination thereof, can be used for interacting with a virtualinterface display by an HMD. Any tracking technique known in the art canbe used, e.g. inside-out tracking, outside-in tracking or a combinationthereof, as described, for example in PCT International ApplicationSerial Nos. PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/61698 andPCT/US2019/015522, which are hereby incorporated in their entirety.

A tracked pointer, a tracked stylus, or another tracked tool or trackedinstrument or a combination thereof can comprise one or more markers. Insome embodiments, the marker can be configured to reflect or emit lightwith a wavelength between 380 nm and 700 nm or any value or range orsubrange therebetween. In some embodiments, the marker can be configuredto reflect or emit light with a wavelength greater than 700 nm. Forexample, the marker can be configured to reflect or emit light with awavelength between 700 nm and 1 mm or any value or range or subrangetherebetween. In some embodiments, the marker can be configured toreflect or emit light with a wavelength less than 380 nm. For example,the marker can be configured to reflect or emit light with a wavelengthbetween 50 nm and 380 nm or any value or range or subrange therebetween.In some embodiments, the marker can be a radiofrequency marker, e.g. anactive marker, an infrared marker, e.g. a retroreflective or passivemarker. The marker can be an optical marker, e.g. an optical marker thatcomprises a geometric pattern. One, two or more markers can be attachedto or integrated into a tracked pointer, a tracked stylus, other trackedtool, other tracked instrument or a combination thereof. A trackedpointer, a tracked stylus, other tracked tool, other tracked instrumentor a combination thereof can also comprise one or more integrated orattached IMUs.

In some embodiments, the system comprises at least one camera, scanner(e.g. 3D scanner, laser scanner), LIDAR system, depth sensor, IMU or acombination thereof integrated into or attached to the head mounteddisplay or other augmented reality display device. In some embodiments,at least one camera, scanner (e.g. 3D scanner, laser scanner), LIDARsystem, depth sensor, IMU or a combination thereof can be separate fromthe head mounted display or other augmented reality display device. Theat least one camera, scanner (e.g. 3D scanner, laser scanner), LIDARsystem, depth sensor, IMU or a combination thereof integrated orattached to the one or more HMDs or other augmented reality displaysystems and/or separate from the one or more HMDs or other augmentedreality display systems can be configured to scan and/or detect apointer, stylus, tool, instrument or a combination thereof; the pointer,stylus, tool, instrument or a combination thereof can be tracked in 3Dspace, e.g. as they are being moved by a user. The tracking can bedirect, e.g. by directly recognizing the instrument, for exampleutilizing a stored shape, surface, and/or 2D or 3D outline data of thepointer, stylus, tool, instrument or combination thereof or a library ofshapes, surfaces, and/or 2D or 3D outline data of one or more pointer,stylus, tool, instrument or combination thereof, using one or morecomputer processors.

Direct Tracking of Tool and/or Instrument for Interaction with VirtualInterface

One or more computer processors can be configured to detect a physicaltool or instrument, e.g. a tracked pointer, a tracked stylus, othertracked tool or tracked instrument, or a combination thereof, using acamera and/or scanner (e.g. a video camera, infrared camera, 3D scanner,laser scanner, LIDAR, imaging system etc.), e.g. in the image or scannerdata, and, optionally, follow and/or track the pointer, stylus, tool,instrument, or combination thereof in real-time or near real-time, forexample within the 3D space included in the image or scanner data. Theone or more camera and/or scanner (e.g. a video camera, infrared camera,3D scanner, laser scanner, LIDAR, imaging system etc.) can be integratedinto or attached to one or more HMD units or can be separate from one ormore HMD units or a combination thereof. Optionally, a pointer, stylus,tool, instrument, or combination thereof included in the image orscanner data can be compared against a database or library of storedshapes, surfaces, and/or 2D or 3D outline data of one or more(optionally different) pointer(s), stylus(s), other tool(s), otherinstrument(s) or combination thereof and can be identified using thedatabase of stored shapes, surfaces, and/or 2D or 3D outline data of theone or more (optionally different) pointer(s), stylus(s), other tool(s),other instrument(s) or combination thereof. Identification of thepointer, stylus, tool, instrument, or combination thereof included inthe image or scanner data can, optionally, facilitate tracking of thepointer, stylus, tool, instrument or combination thereof. The databasecan also comprise information about one or more optical markers,fiducials, fiducial arrays, and/or marker or array configurations.

In some embodiments, a pointer, a surgical tool or instrument cancomprise a unique marker, fiducial array or marker/array configuration.In this manner, one or more computer processors can be configured todetect and/or identify the unique marker, fiducial array or marker/arrayconfiguration associated with that pointer, surgical tool or instrument.If a pointer, tool or instrument has been identified by at least onecomputer processor using any of the foregoing techniques, the at leastone computer processor can associate and/or activate and/or displaycertain functions, e.g. display functions and/or system functionsassociated with that tool. For example, if a pointer has been identifiedby the at least one computer processor, e.g. based on identification ofits unique shape (e.g. relative to a database of tool shapes) or basedon identification of a unique marker, fiducial array or marker/arrayconfiguration, the identification by the at least one computer processorcan trigger or initiate a specific function. For example, during set-upof an augmented reality system for a surgical procedure, a calibrationprocedure can be activated. When a pointer (or other tool or instrument)is identified the system can, for example, automatically display, by anHMD or other augmented reality device, a virtual object, e.g. oneassociated with the calibration procedure. In this example of an ARdisplay calibration, the virtual object can be moved by moving thetracked physical pointer. The movement of the virtual object can becorresponding to the movement of the tracked physical pointer, or it canbe at a different movement ratio between movement of the virtual objectand movement of the tracked physical pointer, e.g. 1:1, 1.5:1, 2.0:1,0.5:1.0, or any other ratio, for example expressed in mm, cm, and/orangular degrees. By moving the tracked physical pointer so that thevirtual object, e.g. a virtual marker ball, is superimposed and/oraligned with a corresponding physical object, e.g. a physical markerball, the system can determine the coordinate difference and/orcoordinate transfer and/or distance, angular movement required tosuperimpose the virtual object onto the physical object; the informationcan be used to move a virtual display, facilitated by at least onecomputer processor, as a means of optimizing the superimposition and/oralignment for a user between virtual display of any virtual objects(e.g. a virtual spine and/or virtual instrument and/or virtual implant)and the corresponding physical objects and/or structures (e.g. acorresponding physical spine and/or a corresponding physical instrumentand/or a corresponding physical implant). The data/information relatedto the coordinate difference and/or coordinate transfer and/or distanceand/or angular movement required to superimpose a virtual object onto acorresponding physical object can be used, by the at least one computerprocessor, as a means of AR system calibration to more closely match avirtual AR display, axis of an AR display, center of an AR display withthe optical axis of the user's eye(s). The data can optionally bewirelessly transmitted and/or received by a first computing system (e.g.communicably connected to a navigation system, a robot, and/or animaging system) and a second (or more) computing system(s) (communicablyconnected to one or more HMDs or other augmented reality displaydevices).

In another example, a system can detect another unique marker, fiducialarray or marker/array configuration associated, for example, withanother instrument, e.g. an awl. The identification of the uniquemarker, fiducial array or marker/array configuration associated with theawl can trigger a display, by an HMD or other augmented reality displaydevice, of a targeting tool for targeting the awl, e.g. superimposedonto a target anatomic structure of the patient. One or more computerprocessors can be configured to detect and/or identify the tip and/oraxis and/or direction of movement of a pointer, stylus, tool, instrumentor combination thereof, for example by detecting a shape, contour and/oroutline, optional identification of the stylus, tool, instrument orcombination thereof used, and optional use of known shape data and/ordimensions of the stylus, tool, instrument or combination thereof.

The one or more computer processors configured to detect a pointer,stylus, other tool, other instrument, or combination thereof using acamera and/or scanner, e.g. in the image or scanner data, and,optionally, configured to follow and/or track the pointer, stylus, othertool, other instrument, or combination thereof in real-time or nearreal-time can be part of a first computing system, for example, a serveror controller or computing unit as shown in FIG. 1, 1180. The one ormore computer processors configured to detect a pointer, stylus, othertool, other instrument, or combination thereof using a camera and/orscanner, e.g. in the image or scanner data, and, optionally, configuredto follow and/or track the pointer, stylus, other tool, otherinstrument, or combination thereof in real-time or near real-time can bepart of a second computing system, e.g. a client or mobile unitintegrated into, attached to or connected via cable to an HMD.Inside-out and/or outside-in tracking techniques can be used by one ormore computer processors for tracking a pointer, stylus, other tool,other instrument, or combination thereof using a camera and/or scanner.

Tracking of Tool and/or Instrument Using Markers for Interaction withVirtual Interface

One or more computer processors can be configured to detect one or moremarkers integrated or attached to one or more physical tool orinstrument, e.g. a tracked pointer, a tracked stylus, other tracked toolor tracked instrument, or a combination thereof, using a camera and/orscanner, e.g. a video camera, infrared camera, 3D scanner, laserscanner, LIDAR, imaging system etc., and, optionally, follow and/ortrack the one or more pointer, stylus, tool, or instrument in real-timeor near real-time, using the one or more markers. The one or more cameraand/or scanner (e.g. a video camera, infrared camera, 3D scanner, laserscanner, LIDAR, imaging system etc.) can be integrated into or attachedto one or more HMD units or can be separate from one or more HMD unitsor a combination thereof.

One or more computer processors can be configured to detect and/oridentify the tip and/or axis and/or direction of movement of a pointer,stylus, other tool, other instrument or combination thereof, using oneor more integrated or attached markers.

The one or more computer processors configured to detect and/or trackone or more markers integrated or attached to one or more pointer,stylus, other tool, other instrument, or combination thereof, using acamera and/or scanner in real-time or near real-time can be part of afirst computing system, for example, a server or controller or computingunit as shown in FIG. 1, 1180. The one or more computer processorsconfigured to detect and/or track one or more markers integrated orattached to one or more pointer, stylus, other tool, other instrument,or combination thereof, using a camera and/or scanner in real-time ornear real-time can be part of a second computing system, e.g. a clientor mobile unit integrated into, attached to or connected via cable to anHMD. Inside-out and/or outside-in tracking techniques can be used by oneor more computer processors for tracking one or more markers integratedinto or attached to a pointer, stylus, tool, instrument, or combinationthereof using a camera and/or scanner. The markers can be any of themarkers described in the specification or known in the art, e.g. activemarkers, passive markers, infrared markers, retroreflective markers,radiofrequency markers, optical markers, e.g. with geometric patterns,bar codes, QR codes, Aruco codes etc.

Virtual Interface Display

One or more HMDs or other augmented reality display systems canoptionally generate a 2D or 3D stereoscopic or non-stereoscopic virtualdisplay or augmented view comprising a virtual interface, e.g.superimposed on a physical patient, physical anatomic structure,physical anatomic landmark and/or physical anatomic surface and/or nearor adjacent to a physical patient, physical anatomic structure, physicalanatomic landmark and/or physical anatomic surface. One or more computerprocessors, e.g. in a first (e.g. a server) or second (e.g. a client)computing system can be configured to generate a 2D or 3D stereoscopicor non-stereoscopic virtual display comprising a virtual interface at apredetermined location and/or orientation relative to one or moreanatomic structures and/or the patient. One or more computer processors,e.g. in a first (e.g. a server) or second (e.g. a client) computingsystem can be configured to generate a 2D or 3D stereoscopic ornon-stereoscopic virtual display comprising a virtual interface, forexample at a predetermined location and/or orientation relative to oneor more markers (e.g. infrared markers, radiofrequency markers, activemarkers, passive markers, optical markers [e.g. with geometric patterns,bar codes, QR codes etc.], LED's etc.) attached to one or more anatomicstructures and/or the patient and/or a fixed structure in the operatingroom. The one or more markers can, for example, be a fiducial arrayattached to one or more bones. One or more computer processors, e.g. ina first (e.g. a server) or second (e.g. a client) computing system canbe configured to generate a 2D or 3D stereoscopic or non-stereoscopicvirtual display comprising a virtual interface at a predeterminedlocation and/or orientation relative to one or more structures in theoperating room, e.g. an OR table, an OR light, an external computermonitor etc. One or more computer processors, e.g. in a first (e.g. aserver) or second (e.g. a client) computing system can be configured togenerate a 2D or 3D stereoscopic or non-stereoscopic virtual displaycomprising a virtual interface at a predetermined location and/ororientation relative to the user's eyes and/or face and/or relative tothe physical HMD or other augmented reality display unit, e.g. thehousing of the HMD unit or the physical display (e.g. combiner,waveguide and/or mirror) of the HMD.

A 2D or 3D stereoscopic or non-stereoscopic virtual display comprising avirtual interface displayed by one or more HMDs or other augmentedreality display systems can comprise, for example, one or more virtualbutton, virtual field, virtual cursor, virtual pointer, virtual slider,virtual trackball, virtual node, virtual numeric display, virtualtouchpad, virtual keyboard, or a combination thereof. The one or moreone or more virtual button, virtual field, virtual cursor, virtualpointer, virtual slider, virtual trackball, virtual node, virtualnumeric display, virtual touchpad, virtual keyboard, or a combinationthereof can be displayed, by the one or more HMDs or other augmentedreality display systems, using one or more computer processors, in 2D,in 3D, or a combination thereof. For example, a virtual slider can be in2D and/or in 3D. A 3D virtual slider can, for example, comprise anactivation or sliding field oriented in x-direction, an activation orsliding field oriented in y-direction, and an activation or slidingfield oriented in z-direction. The system can be configured fordetecting various interactions by a user with the one or more virtualobjects of a virtual interface, for example an interaction via gesturerecognition, gaze recognition, gaze lock, eye tracking, hand tracking,pointer tracking, instrument tracking, tool tracking, or a combinationthereof. For example, a tracked finger, tracked hand, tracked pointer,tracked instrument, tracked tool can interact with the virtualinterface. The interaction can trigger an event message, optionallymanaged by an event handler, and/or a command. The interaction, eventmessage, command or combination thereof can optionally be transmittedand/or received between a first and a second computing system, forexample a first computing system (e.g. communicably connected to anavigation system, a robot, and/or an imaging system) and a second (ormore) computing system(s) (communicably connected to one or more HMDs orother augmented reality display devices.

Collision Detection

In some embodiments, the system can comprise a collision detectionmodule or other interaction module. The collision detection module orother interaction module can be a module separate from other modules,such as an AR visualization module 1150 or an AR display module 1160.The collision detection module or other interaction module can be partof another module, e.g. a submodule of another module, such as an ARvisualization module 1150 or an AR display module 1160.

One or more of the computer processors operating a collision detectionmodule or other interaction module, one or more computer processorsoperating a tracking engine or tracking module 1100, one or morecomputer processors operating an instrument calibration module 1110, oneor more of the computer processors operating a headset calibrationmodule 1120, one or more of the computer processors operating an imagingand navigation module 1130, one or more of the computer processorsoperating an AR wireless networking module 1140, one or more of thecomputer processors operating an AR visualization module 1150, and/orone or more of the computer processors operating an AR display module1160 can be the same.

One or more of the computer processors operating a collision detectionmodule or other interaction module, one or more computer processorsoperating a tracking engine or tracking module 1100, one or morecomputer processors operating an instrument calibration module 1110, oneor more of the computer processors operating a headset calibrationmodule 1120, one or more of the computer processors operating an imagingand navigation module 1130, one or more of the computer processorsoperating an AR wireless networking module 1140, one or more of thecomputer processors operating an AR visualization module 1150, and/orone or more of the computer processors operating an AR display module1160 can be different.

In some embodiments, one or more HMDs or other augmented reality displaysystems and one or more physical tools or physical instruments, e.g. apointer, a stylus, other tools, other instruments, can be tracked, e.g.using inside-out or outside-in tracking. The coordinates, positionand/or orientation of a virtual display comprising a virtual interfacedisplayed by one or more HMDs or other augmented reality display systemscan also be tracked. One or more computer processors can be configured,using one or more collision detection modules, to detect collisionsbetween a gaze (e.g. using gaze tracking, gaze lock), a finger (e.g.using finger/hand tracking), a hand (e.g. using hand tracking), an eye(e.g. using eye tracking), one or more tracked physical tools orphysical instruments, e.g. a tracked pointer, a tracked stylus, othertracked physical tools, other tracked physical instruments, or acombination thereof and a virtual display comprising the virtualinterface, e.g. one or more virtual objects such as virtual button,virtual field, virtual cursor, virtual pointer, virtual slider, virtualtrackball, virtual node, virtual numeric display, virtual touchpad,virtual keyboard, or a combination thereof.

One or more computer processors can use polygon based collisiondetection or detection of other interactions. One or more computerprocessors can use volume based collision detection or detection ofother interactions. One or more computer processors can be configuredwith a predetermined tolerance for a collision detection or detection ofother interactions, e.g. <0.1, <0.5, <1.0, <1.5, <2.0, <3.0, <4.0, <5.0,<10.0 mm, or any other value, and/or <0.1, <0.5, <1.0, <1.5, <2.0, <3.0,<4.0, <5.0, <10.0, <15.0, <20.0 degrees, or any other value. In someembodiments, the tolerance for a collision detection or detection ofother interactions can be selected and/or predetermined, for example, toenable a particular application, e.g. activating or executing a command,moving a virtual slider, selecting a virtual button, etc. The tolerancefor a collision detection or detection of other interactions can be thesame or different for different applications, e.g. an HMD calibration,an instrument calibration, an AR visualization module, e.g. comprisingselection of a predetermined path for a physical tool or instrument.

Different collision detection modules or packages known in the artinclude, for example I-collide [Cohen et al. “I-COLLIDE: An Interactiveand Exact Collision Detection System for Large-Scale Environments,” inThe 1995 ACM International 3D Graphics Conference], V-clip [Mirtich etal. “V-Clip: Fast and Robust Polyhedral Collision Detection,” ACM Trans.Graphics, 17, 3, pp. 177-208], SWIFT [Ehrmann et al. “SWIFT: AcceleratedProximity Queries Between Convex Polyhedra by Multi-Level VoronoiMarching,” Technical report, Computer Science Department, University ofNorth Carolina at Chapel Hill], RAPID [Gottschalk et al. OBB-Tree: AHierarchical Structure for Rapid Interference Detection,” ComputerGraphics SIGGRAPH '96 Proceedings 30, pp. 171-180], V-collide [Hudson etal. “V-COLLIDE: Accelerated Collision Detection for VRML”, inProceedings of the Second Symposium on Virtual Reality ModelingLanguage. California, United States, ACM Press], PQP [Larsen et al.“Fast Proximity Queries With Swept Sphere Volumes” Technical ReportTR99-018, Department of Computer Science, University of North Carolina.SOLID [Bergen et al. “User's Guide to the SOLID Interference DetectionLibrary”; SWIFT [Ehrmann et al. “Accurate and Fast Proximity QueriesBetween Polyhedra Using Surface Decomposition”, Eurographics. ComputerGraphics Forum, 20, 3, or VPS [McNeely et al. “Six Degree-of-FreedomHaptic Rendering Using Voxel Sampling”, SIGGRAPH 99 ConferenceProceedings, Annual Conference Series, pp. 401-408].

In some embodiments, a collision detection module, e.g. I-collide, canutilize convex polyhedra for multi-body collision detection. In someembodiments, a collision detection module such as RAPID can utilizenon-convex models, detecting, for example, detects pair-wise collisions.Some collision detection modules, e.g. V-collide, can be configured todetect multiple body collisions. Some collision detection modules, e.g.PQP, can support non-convex modes and/or can optionally perform distancecomputation and/or tolerance verification queries. Some collisiondetection modules, e.g. SWIFT, can comprise intersection detection,tolerance verification, exact and approximate distance computation,contact determination, or a combination thereof. Some collisiondetection methods, e.g. I-collide, RAPID, PQP and/or SWIFT, can be basedon polygon intersection. Some collision detection packages, e.g. VPS,can utilize voxels and can, for example, detect collisions, performtolerance verification, approximate distances, and determine contactnormal, center of mass, or a combination thereof.

One or more computer processors can be configured to utilize sweep-basedcontinuous collision detection. Sweep-based continuous collisiondetection can use a Time Of Impact (TOI) algorithm to compute potentialcollisions, e.g. for a gaze (e.g. using gaze tracking, gaze lock), afinger (e.g. using finger/hand tracking), a hand (e.g. using handtracking), an eye (e.g. using eye tracking), for one or more trackedphysical pointer, tracked physical tool, tracked physical instrument, ora combination thereof by sweeping its forward trajectory using itscurrent velocity based on the tracking data. If there are contacts withthe virtual display, comprising, for example a virtual interface, e.g.along the moving direction of the one or more tracked physical pointer,tracked physical tool, tracked physical instrument, or a combinationthereof, one or more computer processors can be configured to detect thecollision. In some embodiments, one or more computer processors can beconfigured to compute a time of impact, e.g. for a given movingdirection and/or speed of a tracked physical pointer, physical tool,tracked physical instrument, or combination thereof. The one or morecomputer processors can perform sub steps from that time onwards,computing the velocity after TOI then re-sweep.

One or more computer processors can be configured to utilize speculativecontinuous collision detection. Speculative continuous collisiondetection can operate by increasing a broad-phase axis-aligned minimumbounding box of a tracked physical surgical tool, tracked physicalinstrument or combination thereof, based on the linear and angularmotion of the tracked physical surgical tool, tracked physicalinstrument or combination thereof. The algorithm can be speculativesince it can pick all potential contacts during the next physical step.The contacts can then be fed into a solving program operated by one ormore computer processors, which can ensure that applicable orpredetermined contact constraints can be satisfied. One or more computerprocessors, e.g. on a first computing unit (e.g. a server) and/or asecond computing unit (e.g. a client integrated or connected to an HMDor other augmented reality display device) can operate commerciallyavailable software with one or more integrated collision detectionmodules or programs, e.g. Unity software (Unity Software, Inc.).

Commands and/or Executable Actions Triggered Using Virtual Interface

In some embodiments, a first computing unit, e.g. a server orcontroller, can comprise a collision detection module or a module fordetection of other interactions (e.g. software program). In someembodiments, a second computing unit, e.g. a mobile client (for examplecommunicably connected to or part of one or more HMD or other augmentedreality display units) can comprise a collision detection module or amodule for detection of other interactions, program or software. In someembodiments, for example when a system comprises multiple HMDs or otheraugmented reality display systems, a second, third, fourth, fifth,sixth, etc. computing unit, e.g. a second, third, fourth, fifth, sixth,etc. wireless, mobile client, can comprise a collision detection moduleor a module for detection of other interactions, program or software,for example a collision detection module or a module for detection ofother interactions, program or software for each HMD unit and associatedcomputing unit. In some embodiments, one or more computer processors ofa first computing unit and a second, and/or third, and/or fourth, and/orfifth, and/or sixth, etc. computing unit can operate the same collisiondetection module or module for detection of other interactions. In someembodiments, one or more computer processors of a first computing unitand a second, and/or third, and/or fourth, and/or fifth, and/or sixth,etc. computing unit can operate different collision detection modules ormodules for detection of other interactions.

In some embodiments, one or more computer processors of a firstcomputing unit and a second, and/or third, and/or fourth, and/or fifth,and/or sixth, etc. computing unit can operate the same collisiondetection module or module for detection of other interactions, whichcan be used for the same functions and/or interactions and/or commandsof/with a virtual object displayed as part of a virtual interface. Insome embodiments, one or more computer processors of a first computingunit and a second, and/or third, and/or fourth, and/or fifth, and/orsixth, etc. computing unit can operate the same collision detectionmodule or module for detection of other interactions, which can be usedfor the different functions and/or interactions and/or commands of/witha virtual interface.

In some embodiments, one or more data packets, for example as describedin Table 2, e.g. tracking data of one or more HMDs or other augmentedreality display systems, one or more virtual displays, one or morephysical pointers, physical tools, or physical instruments, one or morephysical implants, one or more robots or robotic arms, can betransmitted from a first computing system wirelessly to a second, and/orthird, and/or fourth, and/or fifth, and/or sixth, etc. computing system,e.g. a second, and/or third, and/or fourth, and/or fifth, and/or sixth,etc. wireless, mobile client integrated into or connected to a first,and/or second, and/or third, and/or fourth, and/or fifth, etc. HMD orother augmented reality display unit. One or more computer processors ofthe second, and/or third, and/or fourth, and/or fifth, and/or sixth,etc. wireless, mobile client can operate one or more collision detectionmodules, e.g. using Unity software (Unity Software, Inc., 30 3^(rd)Street, San Francisco, Calif. 94103), to detect collisions of one ormore tracked physical tools or instruments, e.g. a physical pointer or aphysical stylus or other physical tool or instrument, with a virtualinterface displayed by the one or more HMDs or other augmented realitydisplay systems; the collision(s) of the tracked physical tool orinstrument, or a gaze (e.g. using gaze tracking, gaze lock), a finger(e.g. using finger/hand tracking), a hand (e.g. using hand tracking), aneye (e.g. using eye tracking) with different portions, aspects, fieldsor displays of the virtual interface, e.g. a virtual button, virtualfield, virtual cursor, virtual pointer, virtual slider, virtualtrackball, virtual node, virtual numeric display, virtual touchpad,virtual keyboard, or a combination thereof, can be used to trigger oneor more actions and/or one or more commands, which can, optionally betransmitted from the second, and/or third, and/or fourth, and/or fifth,and/or sixth, etc. wireless, mobile client and be received by the firstcomputing unit for further processing, e.g. execution of commands by oneor more computer processors.

In some embodiments, coordinate and/or tracking data of one or more HMDsor other augmented reality display systems and/or one or more virtualdisplays by one or more HMDs or other augmented reality display systems(see Table 2) can be received by a first computing unit, along withtracking data of one or more physical tools or physical instruments, oneor more physical implants, one or more robots or robotic arms. The firstcomputing unit or system can operate one or more collision detectionmodules to detect collisions of one or more tracked physical tools orinstruments, e.g. a physical pointer or a physical stylus or otherphysical tool or instrument, with the virtual display, e.g. a virtualinterface, displayed by the one or more HMDs or other augmented realitydisplay systems. The collision(s) of the tracked physical tool orinstrument with different portions, aspects, fields or displays of thevirtual interface, e.g. a virtual object such as a virtual button,virtual field, virtual cursor, virtual pointer, virtual slider, virtualtrackball, virtual node, virtual numeric display, virtual touchpad,virtual keyboard, or a combination thereof, can be used to trigger oneor more actions and/or one or more commands, which can be processed byone or more computer processors of the first computing system and whichcan, optionally be transmitted to a second, and/or third, and/or fourth,and/or fifth, and/or sixth, etc. wireless, mobile client connected to orintegrated into a first, second, third, fourth, and/or fifth etc. HMD orother augmented reality display device, optionally triggering commandsand/or actions by one or more computer processors of the second, and/orthird, and/or fourth, and/or fifth, and/or sixth, etc. wireless, mobileclient(s) and one or more connected HMDs or other augmented realitydisplay systems.

In some embodiments, a physical tool or instrument (see Table 2), e.g. atracked pointer, a tracked stylus, a tracked tool, a tracked instrumentor a combination thereof, can be used for interacting with a virtualinterface display by an HMD or other augmented reality display device.In some embodiments, a gaze (e.g. using gaze tracking, gaze lock), afinger (e.g. using finger/hand tracking), a hand (e.g. using handtracking), an eye (e.g. using eye tracking) or a combination thereof canbe used for interacting with a virtual interface display by an HMD orother augmented reality display device. In some embodiments, a collisionor other interaction, e.g. of a tracked physical tool or instrument,with a virtual display, e.g. a virtual object in a virtual userinterface, displayed by a first HMD can be detected by one or morecomputer processors in a first computing system, e.g. a server, and/or asecond computing system, e.g. a client integrated, attached to orconnected to the first HMD (and/or optionally a second, third, fourth,fifth, sixth or more HMD). The collision and/or other interaction canoptionally be used to execute a function and/or to change the appearanceof the virtual display, e.g. virtual interface, e.g. with a colorchange, display of different buttons and/or functions, etc.

Data and/or execution functions related to or triggered by a collisionand/or other interaction of a tracked physical tool or instrument with avirtual display, e.g. a virtual interface, and/or changes in a virtualdisplay, e.g. a virtual interface, triggered by the collision or otherinteraction with the tracked physical tool or instrument for display byone or more HMDs or other augmented reality display systems can begenerated by one or more computer processors in a first computing unit,e.g. a server, and can be transmitted to one or more additionalcomputing units, for example a second, third, fourth, fifth or morecomputing unit, e.g. a client integrated, attached to or connected toone or more HMDs or other augmented reality display systems.

Data and/or execution functions related to or triggered by a collisionand/or other interaction of a tracked physical tool or instrument with avirtual display, e.g. a virtual interface, and/or changes in a virtualdisplay, e.g. a virtual interface, triggered by the collision or otherinteraction with the tracked physical tool or instrument for display byone or more HMDs or other augmented reality display systems can begenerated by one or more computer processors in a second (or first)computing unit, e.g. a client integrated, attached to or connected to afirst HMD, and can be transmitted to a first (or second) computing unit,e.g. a server (e.g. separate from the one or more HMDs or otheraugmented reality display systems). One or more computer processors inthe second (or first) computing unit, e.g. the server, can be configuredto process and/or transmit the data and/or execution functions relatedto or triggered by the collision and/or other interaction of the trackedphysical tool or instrument with the virtual display, e.g. a virtualinterface, and/or changes in the virtual display, e.g. a virtualinterface, triggered by the collision or other interaction with thetracked physical tool or instrument to one or more additional computingunits, for example a second, third, fourth, fifth or more computingunit, e.g. one or more clients integrated, attached to or connected toone or more additional HMDs or other augmented reality display systemsfor display by the one or more additional HMDs or other augmentedreality display systems.

FIG. 3 is a representative, non-limiting example of a tracking system1300, e.g. one or more cameras, including cameras using visible and/orinfrared light, stereoscopic cameras, a Lidar system and/or camera(s), ascanner, for example a 3D scanner, one or more computer systems (CS) CS#1 1310, CS #2 1320, CS #3 1330, each with one or more computerprocessors, one or more robots 1340, e.g. handheld, attached to an ORtable, and/or comprising a robotic arm, one or more imaging systems1350, e.g. a C-arm, 3D C-arm, cone beam CT, spiral CT, MRI, and/or oneor more head mounted displays HMD A 1360 HMD B 1370, e.g. a stereoscopicoptical see-through head mounted display and/or a stereoscopic videosee-through head mounted display. The one or more robots 1340 and/or oneor more imaging systems 1350 can also comprise one or more computersystems (e.g. CS #4, CS #5), for example each with one or more computerprocessors.

A first computer system CS #1 1310 can, for example, reside in a serveror computer, for example located in the operating room. The one or morefirst computer systems CS #1 1310 can also be located in a remotelocation, e.g. outside the operating room, and/or can comprise a cloudcomputing system and/or communicate through a cloud computing system,e.g. through a wired or a wireless connection. One or more second CS #21320, third CS #3 1330, or more computer systems can be integrated into,connected to or attached to one or more head mounted displays HMD A 1360and/or HMD B 1370 or other augmented reality display devices. Thetracking system 1300 can be separate from the one or more head mounteddisplays HMD A 1360 and/or HMD B 1370 or other augmented reality displaydevices. In some embodiments, the tracking system 1300 can be integratedor attached to the one or more head mounted displays HMD A 1360 and/orHMD B 1370 or other augmented reality display devices.

The tracking system 1300 can be configured to track 1410 optionally ahand and/or finger 1380, a pointer, tool, instrument and/or implant1390, optionally a first 1360, second 1370, third, fourth, fifth etc.head mounted display, a patient (not shown), and/or a robot 1340 and/oran imaging system 1350 or a combination thereof, for example with use ofthe first computer system CS #1 1310 and/or optionally with use of asecond computer system CS #2 1320, a third computer CS #3 1330, and/oradditional computer systems.

The tracking system 1300 can be configured to track 1410 optionally ahand and/or finger 1380, a pointer, tool, instrument and/or implant1390, optionally a first 1360, second 1370, third, fourth, fifth etc.head mounted display, a patient (not shown), and/or a robot 1340 and/oran imaging system 1350 or a combination thereof, and to transmit thetracking information 1415 to a first computer system CS #1 1310 and/oroptionally a second computer system CS #2 1320, a third computer CS #31330, and/or additional computer systems.

The second CS #2 1320, third CS #3 1330, and/or additional computersystems can be integrated into, connected to or attached to one or moreHMDs 1360 1370 or other augmented reality display systems. The one ormore additional computer systems can be attached to, connected to orlocated inside a robotic system 1340 and/or an imaging system 1350. Thetracking 1410 can comprise recording one or more coordinates of and/ortracking a hand and/or finger 1380, a pointer, tool, instrument, and/orimplant 1390, and/or a first 1360, second 1370, third, fourth, fifthetc. head mounted display or other augmented reality display device,and/or a patient (not shown) by the first computer system CS #1 1310and/or the second computer system CS #2 1320, and/or the third computerCS #3 1330, and/or additional computer systems, e.g. in a robot 1340 orimaging system 1350, for example using the camera or scanner 1300. Thetracking 1410 can comprise recording one or more coordinates of a handand/or finger 1380, a pointer, tool, instrument, and/or implant 1390,and/or a first 1360, second 1370, third, fourth, fifth etc. head mounteddisplay, and/or the patient (not shown) by the first computer system CS#1 1310 and/or the second computer system CS #2 1320, and/or the thirdcomputer CS #3 1330, and/or additional computer systems, e.g. in a robot1340 or imaging system 1350, for example using the camera or scanner1300. The system can track a hand and/or finger 1380, a pointer, tool,instrument, and/or implant 1390, and/or a first 1360, second 1370,third, fourth, fifth etc. head mounted display, and/or the patient (notshown) by the first computer system CS #1 1310 and/or the secondcomputer system CS #2 1320, and/or the third computer CS #3 1330, and/oradditional computer systems, e.g. in a robot 1340 or imaging system1350, for example using the camera or scanner 1300.

One or more computer processors of the one or more first 1310, second1320, third 1330 etc. computer systems, can be configured to perform oneor more coordinate transformations 1420 and/or to determine a pose 1420of the hand or finger 1380, the one or more pointer, tool, instrument,and/or implant 1390, the first 1360, second 1370, third, fourth, fifthetc. head mounted display, and/or the patient (not shown).

The one or more computer processors of the one or more first 1310,second 1320, third 1330 etc. or additional computer systems, e.g. alsoin a robot 1340 and/or an imaging system 1350, can be configured totransmit and/or receive 1430 information about the finger or hand 1380,or the one or more pointer, tool, instrument, and/or implant 1390, e.g.their pose, information about or for the first 1360, second 1370, third,fourth, fifth etc. head mounted display or other augmented realitydisplay devices, e.g. the pose of the head mounted display(s) or otheraugmented reality display devices or display data for the head mounteddisplay(s) 1360 1370 or other augmented reality display devices, and/orthe patient (not shown), for example using Bluetooth or a WiFi or LiFiwireless access point, e.g. including a transmitter and/or receiver, ora wired connection. The transmission and/or reception 1430 can comprisedata about gaze direction of the wearer of a headset, e.g. a first 1360,second 1370, third, fourth, fifth etc. head mounted display. Thetransmission and/or reception 1430 can be wireless, e.g. using abroadcast or multiple unicast transmissions to and/or from the first1360, second 1370, third, fourth, fifth etc. head mounted display, asdescribed in the specification. The transmission can comprise positionaldata, display data and/or other data. The transmission 1430 can be to orfrom a first computer system 1310 to or from a second computer system1320, to or from a third computer system 1330, to or from a fourth,fifth or more computer system. The transmission can be to or from asecond computer system 1320 to or from a first computer system 1310, toor from a third computer system 1330, to or from a fourth, fifth or morecomputer system. The transmission can be to or from a third computersystem 1330 to or from a first computer system 1310, to or from a secondcomputer system 1320, to or from a fourth, fifth or more computersystem, and so forth.

In any of the embodiments throughout the specification, the transmission1430 can be unidirectional and/or bi-directional, simultaneousuni-directional and/or bi-directional, and/or non-simultaneous, e.g.sequential uni-directional and/or bi-directional, or a combinationthereof between the one or more first 1310, second 1320, third 1330 etc.or additional computer systems, e.g. also integrated into or attached toa robot and/or an imaging system.

In any of the embodiments throughout the specification, a second 1320,third 1330 etc. or additional computer systems, can be integrated into,connected to, and/or attached to and/or separate from one or more HMDs1360, 1370 or other augmented reality display systems, one or morerobotic systems 1340, and/or one or more imaging systems 1350. One ormore computer systems 1310 can be separate from, e.g. standalone, fromone or more HMDs 1360, 1370 or other augmented reality display systems,one or more robotic systems 1340, and/or one or more imaging systems1350.

One or more computer processors of one or more first 1310, second 1320,third 1330 etc. or additional computer systems, e.g. also in a robot1340 and/or an imaging system 1350, can be configured to generate one ormore user interface controls 1440. The one or more user interfacecontrols 1440 can, for example, be virtual interface controls. The oneor more user interface controls 1440 can comprise, for example, one ormore virtual button, virtual field, virtual cursor, virtual pointer,virtual slider, virtual trackball, virtual node, virtual numericdisplay, virtual touchpad, virtual keyboard, or a combination thereof.

One or more computer processors of the one or more first 1310, second1320, third 1330 etc. or additional computer systems, e.g. also in arobot 1340 and/or an imaging system 1350, can be configured to performone or more collision detections 1445, for example between a hand orfinger 1380 and a virtual interface displayed by one or more headmounted displays 1360 1370 or other augmented reality display devices,or a tracked physical pointer, physical tool, physical instrument,physical implant component 1390 or a combination thereof and a virtualinterface displayed by one or more head mounted displays 1360 1370 orother augmented reality display devices; the collision detection can beperformed, for example, using the tracking system 1300 (or a camera orscanner integrated or attached to an HMD or other augmented realitydisplay devices). One or more computer processors of the one or morefirst 1310, second 1320, third 1330 etc. or additional computer systems,e.g. also in a robot 1340 and/or an imaging system 1350, can beconfigured to perform a gesture recognition, for example between a handor finger 1380 and a virtual interface using one or more cameras,optionally integrated or attached to one or more head mounted displays1360 1370. The first computer system 1310 can, for example, be astandalone computer system or a cloud computing system. The second 1320,third 1330 etc. or additional computer systems, can be, for example,integrated into, connected to, and/or attached to and/or separate fromone or more HMDs 1360, 1370 or other augmented reality display systems,one or more robotic systems 1340, and/or one or more imaging systems1350. One or more computer systems, e.g. a first 1310, second 1320,third 1330 etc. or additional computer systems, e.g. also in a robot1340 and/or an imaging system 1350, can be configured to generate and/ortransmit, optionally wirelessly, one or more event message 1450. The oneor more event message can, for example, be an event message relative toa collision detection 1445 between a hand or finger 1380, a trackedphysical pointer, physical tool, physical instrument, physical implantcomponent 1390 or a combination thereof and a virtual interface, e.g.using the tracking system 1300. The one or more event message 1450 canoptionally comprise an event message related to a robot 1340 control,interface, force, and/or position and/or orientation and/or an imagingsystem 1350 control, interface, force, and/or position and/ororientation.

One or more computer systems, e.g. a first 1310, second 1320, third1330, fourth, fifth etc. or additional computer systems, e.g. also in arobot 1340 and/or an imaging system 1350, can comprise an optional eventhandler 1460 configured to handle, manage and/or process one or moreoptional event message 1450.

One or more computer systems, e.g. a first 1310, second 1320, third 1330etc. or additional computer systems, e.g. also in a robot 1340 and/or animaging system 1350, can be configured to initiate and/or process anoptional event action 1470, e.g. executing a command, e.g. based on theevent message 1450 and, optionally, information or data received fromthe event handler 1460. The command can be transmitted through awireless or a wired connection, e.g. to a robot 1340 or an imagingsystem 1350.

In some embodiments, the one or more computer systems configured togenerate an event message 1450 can be the same or different computersystem comprising the event handler 1460 and/or can be the same ordifferent computer system initiating and/or processing the event action1470, e.g. executing a command. In some embodiments, one or morecomputer processors configured to generate an event message 1450 can bethe same or different computer processors comprising the event handler1460 and/or can be the same or different computer processors initiatingand/or processing the event action 1470, e.g. executing a command. Insome embodiments, a first, second, third, fourth, fifth, sixth or morecomputer system can be the same for managing different tasks, or can bedifferent for managing different tasks.

FIG. 4 is a representative, non-limiting example of one or more trackingsystems 1300, e.g. one or more cameras, including cameras using visibleand/or infrared light, stereoscopic cameras, a Lidar system and/orcamera(s), a scanner, for example a 3D scanner, in this exampleintegrated or attached to one or more head mounted displays 1360 1370 orother augmented reality display devices, one or more computer systems(CS) CS #1 1310, CS #2 1320 (e.g. integrated, connected, or attached toa head mounted display 1360 or other augmented reality display device),CS #3 1330 (e.g. integrated, connected, or attached to a head mounteddisplay 1370 or other augmented reality display device), each with oneor more computer processors, one or more robots 1340, e.g. handheld,attached to an OR table, and/or comprising a robotic arm, one or moreimaging systems 1350, e.g. a C-arm, 3D C-arm, cone beam CT, spiral CT,MRI, and/or one or more head mounted displays HMD A 1360 HMD B 1370,e.g. a stereoscopic optical see-through head mounted display and/or astereoscopic video see-through head mounted display, or other augmentedreality display devices. The one or more robots 1340 and/or one or moreimaging systems 1350 can also comprise one or more computer systems(e.g. CS #4, CS #5), for example each with one or more computerprocessors.

A first, optional computer system CS #1 1310 can, for example, reside ina server or computer, for example located in the operating room. The oneor more optional first computer systems CS #1 1310 can also be locatedin a remote location, e.g. outside the operating room, and/or cancomprise a cloud computing system and/or communicate through a cloudcomputing system, e.g. through a wired or a wireless connection. One ormore second CS #2 1320, third CS #3 1330, or more computer systems canbe integrated into, connected to or attached to one or more head mounteddisplays HMD A 1360 and/or HMD B 1370 or other augmented reality displaydevices. The one or more tracking system(s) 1300 can be integrated orattached to the one or more head mounted displays HMD A 1360 and/or HMDB 1370 or other augmented reality display devices. The one or moretracking systems 1300 can be configured to track 1410 optionally a handand/or finger 1380, e.g. of a user, a pointer, tool, instrument and/orimplant 1390, a patient (not shown), and/or a robot 1340 and/or animaging system 1350 or a combination thereof, for example with use of asecond computer system CS #2 1320, a third computer CS #3 1330, and/oradditional computer systems. An optional first computer system CS #11310 can also be used for tracking, including for processing trackinginformation.

The tracking system 1300 can be configured to track 1410 optionally ahand and/or finger 1380, a pointer, tool, instrument and/or implant1390, a patient (not shown), and/or a robot 1340 and/or an imagingsystem 1350 or a combination thereof, and to transmit the trackinginformation 1415 to a second computer system CS #2 1320, a thirdcomputer CS #3 1330, and/or additional computer systems; thetransmission can be wired or wireless. The one or more tracking systems1300 can be integrated into or attached to one or more head mounteddisplays 1360 1370 or other augmented reality display devices. The oneor more computer systems 1320 1330, including one or more computerprocessors, can be integrated into or connected to the one or more headmounted displays 1360 1370 or other augmented reality display devices.The second CS #2 1320, third CS #3 1330, and/or additional computersystems can be integrated into, connected to or attached to one or moreHMDs 1360 1370 or other augmented reality display systems. The one ormore additional computer systems can be attached to, connected to orlocated inside a robotic system 1340 and/or an imaging system 1350.

The tracking 1410 can comprise recording one or more coordinates ofand/or tracking a hand and/or finger 1380, a pointer, tool, instrument,and/or implant 1390, and/or a patient (not shown) by a second computersystem CS #2 1320, and/or a third computer CS #3 1330, and/or additionalcomputer systems, e.g. in a robot 1340 or imaging system 1350, forexample using the camera(s) or scanner(s) 1300. The tracking 1410 cancomprise recording one or more coordinates of a hand and/or finger 1380,a pointer, tool, instrument, and/or implant 1390, and/or the patient(not shown) by the second computer system CS #2 1320, and/or the thirdcomputer CS #3 1330, and/or additional computer systems, e.g. in a robot1340 or imaging system 1350, for example using the camera or scanner1300. The system can track a hand and/or finger 1380, a pointer, tool,instrument, and/or implant 1390, and/or the patient (not shown) by thesecond computer system CS #2 1320, and/or the third computer CS #3 1330,and/or additional computer systems, e.g. in a robot 1340 or imagingsystem 1350, for example using tracking system, for example with acamera or scanner 1300.

One or more computer processors of the one or more first 1310, second1320, third 1330 etc. computer systems, can be configured to perform oneor more coordinate transformations and/or to determine a pose 1420 ofthe hand or finger 1380, the one or more pointer, tool, instrument,and/or implant 1390, and/or optionally the first 1360, second 1370,third, fourth, fifth etc. head mounted display or other augmentedreality display device (e.g. in relationship to a marker attached to apatient), and/or the patient (not shown).

The one or more computer processors of the one or more first, optional,1310, second 1320, third 1330 etc. or additional computer systems, e.g.also in a robot 1340 and/or an imaging system 1350, can be configured totransmit and/or receive 1430 information about the finger or hand 1380,or the one or more pointer, tool, instrument, and/or implant 1390, e.g.their pose, and/or optionally information about or for the first 1360,second 1370, third, fourth, fifth etc. head mounted display, e.g. thepose of the head mounted display(s) (or other augmented reality displaydevice) (e.g. relative to a marker on a patient) or display data for thehead mounted display(s) 1360 1370 or other augmented reality displaydevices, and/or the patient (not shown), for example using a wirelessconnection, e.g. Bluetooth or a wireless WiFi or LiFi access point,optionally including a transmitter and/or receiver, or using a wiredconnection. The transmission and/or reception 1430 can comprise dataabout gaze direction of the wearer of a headset (for example for usewith a gaze cursor) (or data about gaze direction of the user of anotheraugmented reality display devices), e.g. a first 1360, second 1370,third, fourth, fifth etc. head mounted display. The transmission and/orreception 1430 can be wired or wireless, e.g. using a broadcast ormultiple unicast transmissions to and/or from the first 1360, second1370, third, fourth, fifth etc. head mounted display, as described inthe specification. The transmission can comprise positional data,display data and/or other data. The transmission 1430 can be to or froman optional first computer system 1310 to or from a second computersystem 1320, to or from a third computer system 1330, to or from afourth, fifth or more computer system. The transmission can be to orfrom a second computer system 1320 to or from an optional first computersystem 1310, to or from a third computer system 1330, to or from afourth, fifth or more computer system. The transmission can be to orfrom a third computer system 1330 to or from an optional first computersystem 1310, to or from a second computer system 1320, to or from afourth, fifth or more computer system, and so forth.

In any of the embodiments throughout the specification, the transmission1430 can be unidirectional and/or bi-directional, simultaneousuni-directional and/or bi-directional, and/or non-simultaneous, e.g.sequential uni-directional and/or bi-directional, or a combinationthereof between the one or more first (optional) 1310, second 1320,third 1330 etc. or additional computer systems, e.g. also integratedinto or attached to a robot and/or an imaging system.

In any of the embodiments throughout the specification, a second 1320,third 1330 etc. or additional computer systems, can be integrated into,connected to, and/or attached to and/or separate from one or more HMDs1360, 1370 or other augmented reality display systems, one or morerobotic systems 1340, and/or one or more imaging systems 1350. One ormore computer systems 1310 can be separate from, e.g. standalone, fromone or more HMDs 1360, 1370 or other augmented reality display systems,one or more robotic systems 1340, and/or one or more imaging systems1350.

One or more computer processors of one or more first 1310, second 1320,third 1330 etc. or additional computer systems, e.g. also in a robot1340 and/or an imaging system 1350, can be configured to generate one ormore user interface controls 1440. The one or more user interfacecontrols 1440 can, for example, be virtual interface controls. The oneor more user interface controls 1440 can comprise, for example, one ormore virtual button, virtual field, virtual cursor, virtual pointer,virtual slider, virtual trackball, virtual node, virtual numericdisplay, virtual touchpad, virtual keyboard, or a combination thereof.

One or more computer processors of the one or more first (optional)1310, second 1320, third 1330 etc. or additional computer systems, e.g.also in a robot 1340 and/or an imaging system 1350, can be configured toperform one or more collision detections 1445, for example between ahand or finger 1380 and a virtual interface displayed by one or morehead mounted displays 1360 1370 or other augmented reality displaydevices, or a tracked physical pointer, physical tool, physicalinstrument, physical implant component 1390 or a combination thereof anda virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices; the collision detectioncan be performed, for example, using the tracking system 1300. One ormore computer processors of the one or more first 1310, second 1320,third 1330 etc. or additional computer systems, e.g. also in a robot1340 and/or an imaging system 1350, can be configured to perform agesture recognition, for example between a hand or finger 1380 and avirtual interface, displayed by one or more head mounted displays 13601370, using one or more cameras, optionally integrated or attached toone or more head mounted displays 1360 1370. The one or more cameras canbe part of the one or more tracking systems 1300. The optional firstcomputer system 1310 can, for example, be a standalone computer systemor a cloud computing system. The second 1320, third 1330 etc. oradditional computer systems, can be, for example, integrated into,connected to, and/or attached to and/or separate from one or more HMDs1360, 1370 or other augmented reality display systems, one or morerobotic systems 1340, and/or one or more imaging systems 1350.

One or more computer systems, e.g. a first 1310, second 1320, third 1330etc. or additional computer systems, e.g. also in a robot 1340 and/or animaging system 1350, can be configured to generate and/or transmit,optionally wirelessly, one or more event message 1450. The one or moreevent message can, for example, be an event message relative to acollision detection 1445 between a hand or finger 1380, a trackedphysical pointer, physical tool, physical instrument, physical implantcomponent 1390 or a combination thereof and a virtual interface, e.g.detected using the tracking system 1300. The one or more event message1450 can optionally comprise an event message related to a robot 1340control, interface, force, and/or position and/or orientation and/or animaging system 1350 control, interface, force, and/or position and/ororientation.

One or more computer systems, e.g. a first 1310, second 1320, third1330, fourth, fifth etc. or additional computer systems, e.g. also in arobot 1340 and/or an imaging system 1350, can comprise an optional eventhandler 1460 configured to handle, manage and/or process one or moreoptional event message 1450.

One or more computer systems, e.g. a first 1310, second 1320, third 1330etc. or additional computer systems, e.g. also in a robot 1340 and/or animaging system 1350, can be configured to initiate and/or process anoptional event action 1470, e.g. executing a command, e.g. based on theevent message 1450 and, optionally, information or data received fromthe event handler 1460. The command can be transmitted through awireless or a wired connection, e.g. to a robot 1340 or an imagingsystem 1350.

In some embodiments, the one or more computer systems configured togenerate an event message 1450 can be the same or different computersystem comprising the event handler 1460 and/or can be the same ordifferent computer system initiating and/or processing the event action1470, e.g. executing a command. In some embodiments, one or morecomputer processors configured to generate an event message 1450 can bethe same or different computer processors comprising the event handler1460 and/or can be the same or different computer processors initiatingand/or processing the event action 1470, e.g. executing a command. Insome embodiments, a first, second, third, fourth, fifth, sixth or morecomputer system can be the same for managing different tasks, or can bedifferent for managing different tasks.

FIG. 5 is a representative, non-limiting example of one or more trackingsystems 1300, e.g. one or more cameras, including cameras using visibleand/or infrared light, stereoscopic cameras, a Lidar system and/orcamera(s), a scanner, for example a 3D scanner, in this exampleintegrated or attached to one or more head mounted displays 1360 1370 orother augmented reality display devices, one or more computer systems(CS) CS #1 1510 (e.g. integrated, connected, or attached to a headmounted display 1360), CS #2 1520 (e.g. integrated, connected, orattached to a head mounted display 1360), each with one or more computerprocessors, one or more robots 1530, e.g. handheld, attached to an ORtable, and/or comprising a robotic arm, one or more imaging systems1540, e.g. a C-arm, 3D C-arm, cone beam CT, spiral CT, MRI, and/or oneor more head mounted displays HMD A 1360 HMD B 1370, e.g. a stereoscopicoptical see-through head mounted display and/or a stereoscopic videosee-through head mounted display, or other augmented reality displaydevices. The one or more robots 1530 and/or one or more imaging systems1540 can also comprise one or more computer systems (e.g. CS #3, CS #4),for example each with one or more computer processors.

One or more first CS #1 1510, second CS #2 1520, or more computersystems can be integrated into, connected to or attached to one or morehead mounted displays HMD A 1360 and/or HMD B 1370. The one or moretracking system(s) 1300 can be integrated or attached to the one or morehead mounted displays HMD A 1360 and/or HMD B 1370 or other augmentedreality display devices.

The one or more tracking systems 1300 can be configured to track 1410optionally a hand and/or finger 1380, e.g. of a user a pointer, tool,instrument and/or implant 1390, a patient (not shown), and/or a robot1530 and/or an imaging system 1540 or a combination thereof, for examplewith use of a first computer system 1510, a second computer 1520, and/oradditional computer systems.

The tracking system 1300 can be configured to track 1410 optionally ahand and/or finger (e.g. of a user) 1380, a pointer, tool, instrumentand/or implant 1390, a patient (not shown), and/or a robot 1530 and/oran imaging system 1540 or a combination thereof, and to transmit thetracking information 1415 to a first computer system CS #1 1510, asecond computer CS #2 1520, and/or additional computer systems; thetransmission can be wired or wireless. The one or more tracking systems1300 can be integrated into or attached to one or more head mounteddisplays 1360 1370 or other augmented reality display devices. The oneor more computer systems 1510 1520, including one or more computerprocessors, can be integrated into or connected to the one or more headmounted displays 1360 1370 or other augmented reality display devices.The first CS #1 1510, second CS #2 1520, and/or additional computersystems can be integrated into, connected to or attached to one or moreHMDs 1360 1370 or other augmented reality display systems. The one ormore additional computer systems can be attached to, connected to orlocated inside a robotic system 1530 and/or an imaging system 1540.

The tracking 1410 can comprise recording one or more coordinates ofand/or tracking a hand and/or finger 1380, a pointer, tool, instrument,and/or implant 1390, and/or a patient (not shown) by a first computersystem CS #1 1510, and/or a second computer CS #2 1520, and/oradditional computer systems, e.g. in a robot 1530 or imaging system1540, for example using the camera(s) or scanner(s) 1300. The tracking1410 can comprise recording one or more coordinates of a hand and/orfinger 1380, e.g. of a user, a pointer, tool, instrument, and/or implant1390, and/or the patient (not shown) by the first computer system CS #11510, and/or the second computer CS #2 1520, and/or additional computersystems, e.g. in a robot 1530 or imaging system 1540, for example usingthe camera or scanner 1300. The system can track a hand and/or finger1380, a pointer, tool, instrument, and/or implant 1390, and/or thepatient (not shown) by the first computer system CS #1 1510, and/or thesecond computer CS #2 1520, and/or additional computer systems, e.g. ina robot 1530 or imaging system 1540, for example using the trackingsystem with, for example, a camera or scanner 1300.

One or more computer processors of the one or more first 1510, second1520, etc. computer systems, can be configured to perform one or morecoordinate transformations 1420 and/or to determine a pose 1420 of thehand or finger 1380, the one or more pointer, tool, instrument, and/orimplant 1390, and/or optionally the first 1360, second 1370, third,fourth, fifth etc. head mounted display (e.g. in relationship to amarker attached to a patient), and/or the patient (not shown).

The one or more computer processors of the one or more first 1510,second 1520, and/or additional computer systems, e.g. also in a robot1530 and/or an imaging system 1540, can be configured to transmit and/orreceive 1430 information about the finger or hand 1380, e.g. of a user,or the one or more pointer, tool, instrument, and/or implant 1390, e.g.their pose, and/or optionally information about or for the first 1360,second 1370, third, fourth, fifth etc. head mounted display, e.g. thepose of the head mounted display(s) (e.g. relative to a marker on apatient) or display data for the head mounted display(s) 1360 1370 orother augmented reality display devices, and/or the patient (not shown),for example using a wireless connection, e.g. Bluetooth or a wirelessWiFi or LiFi access point, optionally including a transmitter and/orreceiver, or using a wired connection. The transmission and/or reception1430 can comprise data about gaze direction of the wearer of a headset(for example for use with a gaze cursor), e.g. a first 1360, second1370, third, fourth, fifth etc. head mounted display or other augmentedreality display devices. The transmission and/or reception 1430 can bewired or wireless, e.g. using a broadcast or multiple unicasttransmissions to and/or from the first 1360, second 1370, third, fourth,fifth etc. head mounted display or other augmented reality displaydevices, as described in the specification. The transmission cancomprise positional data, display data and/or other data. Thetransmission 1430 can be to or from a first computer system 1510 to orfrom a second computer system 1520, to or from a third computer system,e.g. part of or coupled to a robot 1530, to or from a fourth, fifth ormore computer system. The transmission can be to or from a secondcomputer system 1520 to or from a first computer system 1510, to or froma third computer system 1530, to or from a fourth, fifth or morecomputer system. The transmission can be to or from a third computersystem 1530 to or from a first computer system 1510, to or from a secondcomputer system 1520, to or from a fourth, fifth or more computersystem, and so forth.

In any of the embodiments throughout the specification, the transmission(and/or reception) 1430 can be unidirectional and/or bi-directional,simultaneous uni-directional and/or bi-directional, and/ornon-simultaneous, e.g. sequential uni-directional and/or bi-directional,or a combination thereof between the one or more first 1510, second1320, third 1330 etc. or additional computer systems, e.g. alsointegrated into or attached to a robot and/or an imaging system.

In any of the embodiments throughout the specification, a first 1510,second 1520 etc. or additional computer systems, can be integrated into,connected to, and/or attached to and/or separate from one or more HMDs1360, 1370 or other augmented reality display systems, one or morerobotic systems 1530, and/or one or more imaging systems 1540. One ormore computer systems can be separate from, e.g. standalone, from one ormore HMDs 1360, 1370 or other augmented reality display systems, one ormore robotic systems 1530, and/or one or more imaging systems 1540.

One or more computer processors of one or more first 1510, second 1520,or additional computer systems, e.g. also in a robot 1530 and/or animaging system 1540, can be configured to generate one or more userinterface controls 1440. The one or more user interface controls 1440can, for example, be virtual interface controls. The one or more userinterface controls 1440 can comprise, for example, one or more virtualbutton, virtual field, virtual cursor, virtual pointer, virtual slider,virtual trackball, virtual node, virtual numeric display, virtualtouchpad, virtual keyboard, or a combination thereof.

One or more computer processors of the one or more first 1510, second1520, or additional computer systems, e.g. also in a robot 1530 and/oran imaging system 1540, can be configured to perform one or morecollision detections 1445, for example between a hand or finger 1380 anda virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices, or a tracked physicalpointer, physical tool, physical instrument, physical implant component1390 or a combination thereof and a virtual interface displayed by oneor more head mounted displays 1360 1370; the collision detection can beperformed, for example, using the tracking system 1300. One or morecomputer processors of the one or more first 1510, second 1520, oradditional computer systems, e.g. also in a robot 1530 and/or an imagingsystem 1540, can be configured to perform a gesture recognition, forexample between a hand or finger 1380 and a virtual interface, displayedby one or more head mounted displays 1360 1370, using one or morecameras, optionally integrated or attached to one or more head mounteddisplays 1360 1370. The one or more cameras can be part of the one ormore tracking systems 1300. The first 1510, second 1520 or additionalcomputer systems, can be, for example, integrated into, connected to,and/or attached to and/or separate from one or more HMDs 1360, 1370 orother augmented reality display systems, one or more robotic systems1530, and/or one or more imaging systems 1540.

One or more computer systems, e.g. a first 1510, second 1520, oradditional computer systems, e.g. also in a robot 1530 and/or an imagingsystem 1540, can be configured to generate and/or transmit, optionallywirelessly, one or more event message 1450. The one or more eventmessage can, for example, be an event message relative to a collisiondetection 1445 between a hand or finger 1380, a tracked physicalpointer, physical tool, physical instrument, physical implant component1390 or a combination thereof and a virtual interface, e.g. detectedusing the tracking system 1300. The one or more event message 1450 canoptionally comprise an event message related to a robot 1530 control,interface, force, and/or position and/or orientation and/or an imagingsystem 1540 control, interface, force, and/or position and/ororientation.

One or more computer systems, e.g. a first 1510, second 1520, third,fourth, fifth etc. or additional computer systems, e.g. also in a robot1530 and/or an imaging system 1540, can comprise an optional eventhandler 1460 configured to handle, manage and/or process one or moreoptional event message 1450.

One or more computer systems, e.g. a first 1510, second 1520, third oradditional computer systems, e.g. also in a robot 1530 and/or an imagingsystem 1540, can be configured to initiate and/or process an optionalevent action 1470, e.g. executing a command, e.g. based on the eventmessage 1450 and, optionally, information or data received from theevent handler 1460. The command can be transmitted through a wireless ora wired connection, e.g. to a robot 1530 or an imaging system 1540.

In some embodiments, the one or more computer systems configured togenerate an event message 1450 can be the same or different computersystem comprising the event handler 1460 and/or can be the same ordifferent computer system initiating and/or processing the event action1470, e.g. executing a command. In some embodiments, one or morecomputer processors configured to generate an event message 1450 can bethe same or different computer processors comprising the event handler1460 and/or can be the same or different computer processors initiatingand/or processing the event action 1470, e.g. executing a command. Insome embodiments, a first, second, third, fourth, fifth, sixth or morecomputer system can be the same for managing different tasks, or can bedifferent for managing different tasks.

AR System Architecture for Use with Multiple Devices Including One orMore HMD

In some embodiments, a system can comprise at least one head mounteddisplay or augmented reality display device, at least one camera orscanning device, wherein the at least one camera or scanning device canbe configured to track real-time information of the at least one headmounted display or augmented reality display device, of at least oneanatomic structure of a patient, and of at least one physical surgicaltool or physical surgical instrument, a first computing systemcomprising one or more computer processors, wherein the first computingsystem can be configured to obtain the real-time tracking information ofthe at least one head mounted display or augmented reality displaydevice, the at least one anatomic structure of a patient, and the atleast one physical surgical tool or physical surgical instrument,wherein the first computing system can be configured for wirelesstransmission of the real-time tracking information of the at least onehead mounted display, the at least one anatomic structure of thepatient, and the at least one physical surgical tool or physicalsurgical instrument, a second computing system comprising one or morecomputer processors, wherein the second computing system can beconfigured for wireless reception of the real-time tracking informationof the at least one head mounted display or augmented reality displaydevice, the at least one anatomic structure of the patient, and the atleast one physical surgical tool or physical surgical instrument,wherein the second computing system can be configured to generate anaugmented or 3D stereoscopic view, wherein the augmented or stereoscopicview can comprise a 3D representation of the at least one trackedphysical surgical tool or physical surgical instrument, and wherein theat least one head mounted display or augmented reality display devicecan be configured to display the 3D stereoscopic view or augmented view.In some embodiments, the one or more computer processors of the secondcomputing system can generate the 3D stereoscopic view for a view angleof the head mounted display relative to the at least one anatomicstructure of the patient using the real-time tracking information of theat least one head mounted display. In some embodiments, the real-timetracking information can comprise tracking information of multiple headmounted displays. In some embodiments, the real-time trackinginformation can comprise a head mounted display or augmented realitydisplay device specific label or tag for each head mounted display, orthe real-time tracking information can be labeled for each tracked headmounted display or augmented reality display device. In someembodiments, the wireless transmission can be a multicast or broadcasttransmission to the multiple head mounted displays. In some embodiments,the real-time tracking information can comprise tracking information oftwo or more head mounted displays or augmented reality display devices.In some embodiments, the two or more head mounted displays or augmentedreality display devices are located in different locations. In someembodiments, the real-time tracking information can comprise a headmounted display or augmented reality display device label for each headmounted display or augmented reality display device, wherein each headmounted display or augmented reality display device has a differentlabel. In some embodiments, the real-time tracking information can belabeled for each tracked head mounted display or augmented realitydisplay device. In some embodiments, one or more computer processors ofa second computing system can generate a 3D stereoscopic view for aninterpupillary distance adjusted for a user wearing the head mounteddisplay. In some embodiments, the second computing system can beintegrated with the at least one head mounted display. In someembodiments, the second computing system can be separate from the atleast one head mounted display and is connected to a display unit of theat least one head mounted display using at least one cable. In someembodiments, the wireless transmission, the wireless reception, or bothcomprise a WiFi signal, a LiFi signal, a Bluetooth signal, aradiofrequency signal or a combination thereof. In some embodiments, acamera or scanning device is separate from at least one head mounteddisplay. In some embodiments, a camera or scanning device can beintegrated or attached to at least one head mounted display. In someembodiments, the wireless transmission can comprise sending data packetscomprising the real-time tracking information of at least one headmounted display, at least one augmented reality display device, at leastone anatomic structure of a patient, and at least one physical surgicaltool or physical surgical instrument, at a rate of 20 Hz or greater. Insome embodiments, a wireless reception can comprise receiving datapackets comprising the real-time tracking information of the at leastone head mounted display, the at least one augmented reality displaydevice, the at least one anatomic structure of a patient, and the atleast one physical surgical tool or physical surgical instrument, at arate of 20 Hz or greater. In some embodiments, the real-time trackinginformation comprises one or more coordinates, e.g. for wirelesstransmission and/or reception, e.g. to and from a first and secondcomputing system. In some embodiments, the one or more coordinates cancomprise coordinates of the at least one anatomic structure of thepatient. In some embodiments, the one or more coordinates can comprisecoordinates of the at least one physical surgical tool or physicalsurgical instrument. In some embodiments, the one or more coordinatescan comprise coordinates of the at least one head mounted display oraugmented reality display device. In some embodiments, at least one headmounted display can comprise at least one optical see-through headmounted display. In some embodiments, at least one head mounted displaycan comprise at least one video see-through head mounted display. Insome embodiments, at least one camera or scanning device can comprise alaser scanner, a time-of-flight 3D laser scanner, a structured-light 3Dscanner, a hand-held laser scanner, a LIDAR scanner, a time-of-flightcamera, a depth camera, a video system, a stereoscopic camera system, acamera array, or a combination thereof. In some embodiments, a systemcan comprise at least one inertial measurement unit. In someembodiments, the at least one inertial measurement unit can beintegrated or attached to the at least one physical surgical tool orphysical surgical instrument. In some embodiments, the at least oneinertial measurement unit can be integrated or attached to the at leastone anatomic structure of the patient. In some embodiments, the at leastone inertial measurement unit can be integrated or attached to the atleast one head mounted display or augmented reality display device. Insome embodiments, the real-time tracking information of the at least onehead mounted display or augmented reality display device can comprisesinformation from the at least one inertial measurement unit. In someembodiments, a second computing system is communicatively coupled to theat least one head mounted display.

In some embodiments, a system can comprise two or more head mounteddisplays or augmented reality display devices, at least one camera orscanning device, wherein the at least one camera or scanning device canbe configured to track real-time information of the at least two or morehead mounted displays or augmented reality display devices, of at leastone anatomic structure of a patient, and of at least one physicalsurgical tool or physical surgical instrument, a first computing systemcomprising one or more computer processors, wherein the first computingsystem can be configured to obtain real-time tracking information of atleast one anatomic structure of a patient, of at least one physicalsurgical tool or physical surgical instrument, and of the two or morehead mounted displays or augmented reality devices, wherein the trackinginformation of the two or more head mounted displays or augmentedreality display devices can be labeled for each of the two or more headmounted displays or augmented reality display devices, wherein the firstcomputing system can be configured for wireless transmission of thereal-time tracking information of the at least one anatomic structure ofthe patient, the tracking information of the at least one physicalsurgical tool or physical surgical instrument, and the labeled trackinginformation of the two or more head mounted displays or augmentedreality display devices, a second computing system, wherein the secondcomputing system can be configured for wireless reception of thereal-time tracking information of the at least one anatomic structure ofthe patient, the tracking information of the at least one physicalsurgical tool or physical surgical instrument, and the labeled trackinginformation of the first of the two or more head mounted displays oraugmented reality display devices, wherein the second computing systemcan be configured to generate a first 3D stereoscopic display specificfor a first viewing perspective of the first head mounted display oraugmented reality display device using the labeled tracking informationof the first head mounted display or augmented reality display device,wherein the first head mounted display or augmented reality displaydevice can be configured to display the 3D stereoscopic display oraugmented view, a third computing system, wherein the third computingsystem can be configured for wireless reception of the real-timetracking information of the at least one anatomic structure of thepatient, the tracking information of the at least one physical surgicaltool or physical surgical instrument, and the labeled trackinginformation of the second of the two or more head mounted displays oraugmented reality display devices, wherein the third computing systemcan be configured to generate a second 3D stereoscopic display oraugmented view specific for a second viewing perspective of the secondhead mounted display or augmented reality display device using thelabeled tracking information of the second head mounted display oraugmented reality display devices, and wherein the first and secondstereoscopic displays or augmented reality display devices can comprisea 3D representation of the at least one physical surgical tool orphysical surgical instrument.

AR Guidance of Surgical Robots

In some embodiments of the disclosure, a system can comprise at leastone head mounted display, a robot, wherein the robot can comprise an endeffector, a first computing system comprising one or more computerprocessors, wherein the first computing system can be in communicationwith the robot, a second computing system comprising one or morecomputer processors, wherein the second computing system can be incommunication with the at least one head mounted display, wherein thesecond computing system can configured to display, by the at least onehead mounted display, a virtual user interface comprising at least onevirtual object, wherein the second computing system can be configured togenerate a command based at least in part on at least one interactionwith the at least one virtual object displayed in the virtual userinterface, wherein the second computing system can be configured totransmit the command to the first computing system using wirelesstransmission, wherein the command can be configured to cause the firstcomputing system to control the robot for movement, activation,operation, de-activation, or any combination thereof, of a robotcomponent, a robot motor, a robot actuator, a robot drive, a robotcontroller, a robot hydraulic system, a robot piezoelectric system, arobot switch, the end effector, or any combination thereof. In someembodiments, the command can be configured to control the end effectorwithin a predetermined operating boundary, a predetermined operatingrange, a predetermined operating zone, or a predetermined operatingvolume. In some embodiments, the first computing system can be connectedto the robot by wire, or the first computing system can be connected tothe robot by wireless connection. In some embodiments, the secondcomputing system can be connected to the at least one head mounteddisplay by wire, or the second computing system can be connected to theat least one head mounted display by wireless connection. In someembodiments, the second computing system can be configured to display,by the at least one head mounted display, a representation of apredetermined operating boundary, a predetermined operating range, apredetermined operating zone, or a predetermined operating volume of theend effector or an expected outcome following the movement, activation,operation, de-activation or a combination thereof of the robotcomponent, robot motor, robot actuator, robot drive, robot controller,robot hydraulic system, robot piezoelectric system, robot switch, theend effector, or any combination thereof. In some embodiments, the endeffector can comprise a physical surgical tool or a physical surgicalinstrument.

In some embodiments, a first computing system can be configured toobtain real-time tracking information of a component of the robot, anend effector, a target object, a target anatomic structure of a patient,at least one head mounted display, a physical tool, a physicalinstrument, a physical implant, a physical object, or any combinationthereof. In some embodiments, a second computing system can beconfigured to obtain real-time tracking information of a component of arobot, an end effector, a target object, a target anatomic structure ofa patient, at least one head mounted display, a physical tool, aphysical instrument, a physical implant, a physical object, or anycombination thereof. In some embodiments, the first computing system canbe configured to obtain real-time tracking information of a physicaltool, a physical instrument, or any combination thereof coupled to therobot. In some embodiments, the second computing system can beconfigured to obtain real-time tracking information of a physical tool,a physical instrument, or any combination thereof coupled to the robot.In some embodiments, the first computing system can be configured towirelessly transmit the real-time tracking information of the componentof the robot, the end effector, a target object, a target anatomicstructure of a patient, the at least one head mounted display, aphysical tool, a physical instrument, a physical implant, a physicalobject, or any combination thereof. In some embodiments, the secondcomputing system can be configured to wirelessly transmit the real-timetracking information of the component of the robot, the end effector, atarget object, a target anatomic structure of a patient, the at leastone head mounted display, a physical tool, a physical instrument, aphysical implant, a physical object, or any combination thereof.

In some embodiments, a second computing system can be configured fordisplaying, by the at least one head mounted display, a 3D stereoscopicview. In some embodiments, the 3D stereoscopic view can be superimposedonto an anatomic structure of a patient. In some embodiments, the 3Dstereoscopic view can comprise a predetermined trajectory of the endeffector, a representation of a predetermined operating boundary of theend effector, a representation of a predetermined operating range of theend effector, a representation of a predetermined operating zone of theend effector, a representation of a predetermined operating volume ofthe end effector, or a combination thereof. In some embodiments, the 3Dstereoscopic view can comprise a predetermined trajectory of the endeffector, a representation of a predetermined operating boundary of theend effector, a representation of a predetermined operating range of theend effector, a representation of a predetermined operating zone of theend effector, a representation of a predetermined operating volume ofthe end effector or a combination thereof following the movement,activation, operation, de-activation or combination thereof of the robotcomponent, robot motor, robot actuator, robot drive, robot controller,robot hydraulic system, robot piezoelectric system, robot switch, theend effector or any combination thereof.

In some embodiments, a first computing system, a second computingsystem, or both can be configured to turn on or turn off the display ofthe virtual user interface. In some embodiments, a wireless transmissioncan comprise a Bluetooth signal, WiFi signal, LiFi signal, aradiofrequency signal, a microwave signal, an ultrasound signal, aninfrared signal, an electromagnetic wave or any combination thereof.

In some embodiments, a 3D stereoscopic view can comprise a predeterminedtrajectory of an end effector, a representation of a predeterminedoperating boundary of the end effector, a representation of apredetermined operating range of the end effector, a representation of apredetermined operating zone of the end effector, a representation of apredetermined operating volume of the end effector or a combinationthereof prior to executing a command. In some embodiments, a the systemcan comprise two or more head mounted displays, wherein the wirelesstransmission can be a multicast, broadcast transmission or anycombination thereof. In some embodiments, a virtual object displayed bythe HMD can comprise one or more virtual button, virtual field, virtualcursor, virtual pointer, virtual slider, virtual trackball, virtualnode, virtual numeric display, virtual touchpad, virtual keyboard, or acombination thereof. In some embodiments, an interaction with a virtualinterface can comprise a collision detection between a physical objectand the at least one virtual object. In some embodiments, an interactioncan be a collision detection between a user's finger and at least onevirtual object displayed by an HMD.

In some embodiments, an interaction can comprise a collision detectionbetween a tracked pointer, tracked tool, tracked instrument, or acombination thereof and at least one virtual object displayed by theHMD. In some embodiments, an interaction with a virtual object displayedby the HMD can comprise a gesture recognition, gaze recognition, gazelock, hand tracking, eye tracking or a combination thereof. In someembodiments, hand tracking, eye tracking and voice control can be used,e.g. also without interaction with a virtual object displayed by theHMD.

In some embodiments, an end effector can comprise a scalpel, a saw, acutting tool, a wire, a needle, a pin, a drill, a burr, a mill, areamer, an impactor, a broach, a laser, a radiofrequency device, athermocoagulation device, a cryoablation device, a radioactive probe, aradioactivity emitting device, a pulsed energy emitting device, anultrasonic energy emitting device, a microwave energy emitting device ora combination thereof. In some embodiments, a command can comprise asubcommand, wherein the subcommand is configured to execute an accept orcancel function of the command.

In some embodiments, a system can comprise at least one head mounteddisplay, a robot, wherein the robot can comprise an end effector, afirst computing system comprising one or more computer processors,wherein the first computing system can be in communication with therobot, a second computing system comprising one or more computerprocessors, wherein the second computing system can be in communicationwith the at least one head mounted display, wherein the second computingsystem can be configured to display, by the at least one head mounteddisplay, a virtual user interface comprising at least one virtualobject, wherein the second computing system can be configured togenerate an event message based at least in part on at least oneinteraction with the at least one virtual object displayed in thevirtual user interface, wherein the second computing system can beconfigured to transmit the event message to the first computing systemusing wireless transmission, wherein the second computing system can beconfigured to generate a command based on the event message, wherein thecommand can be configured to cause the first computing system to controlthe robot for movement, activation, operation, de-activation, or anycombination thereof, of a robot component, a robot motor, a robotactuator, a robot drive, a robot controller, a robot hydraulic system, arobot piezoelectric system, a robot switch, the end effector, or acombination thereof.

AR Guidance of Imaging Systems and/or Image Acquisitions

In some embodiments, preparing an image acquisition by an imaging systemin a patient (prior to the actual image acquisition) can comprisetracking, by at least one computer processor, one or more components ofthe imaging system in real time, optionally obtaining, by the at leastone computer processor, information about a geometry of one or morecomponents of the imaging system, information about a position,orientation and/or geometry of the image acquisition, information aboutone or more image acquisition parameters, or a combination thereof,generating, by the at least one computer processor, a 3D representationof a surface, a volume or combination thereof, wherein the 3Drepresentation of the surface, the volume or combination thereof can beat least in part derived from or based on the information about thegeometry of the one or more components of the imaging system,information about the position, orientation and/or geometry of the imageacquisition, information about the one or more image acquisitionparameters, or a combination thereof, generating, by the at least onecomputer processor, a 3D stereoscopic view or an augmented view, whereinthe 3D stereoscopic view or augmented view can comprise the 3Drepresentation of the surface, volume or combination thereof, optionallydisplaying, by an augmented reality display device, e.g. a head mounteddisplay, a tablet, an iPad, an iPhone or other AR display device, the 3Dstereoscopic view or augmented view at a position defined relative tothe one or more components of the imaging system or at a positiondefined relative to the patient, wherein the position and orientation ofthe 3D stereoscopic view or augmented view can be determined based onthe real time tracking information of the imaging system, and thenacquiring, by the imaging system, 2D, 3D, or 2D and 3D imaging data ofthe patient within the 3D representation of the surface, volume orcombination thereof or at the location of the 3D representation of thesurface, volume or combination thereof.

In some embodiments, the 3D representation of the surface, volume orcombination thereof does not contain imaging data from the patient. Insome embodiments, the 3D representation of the surface, volume orcombination thereof can comprise imaging data from the patient from aprior image acquisition, preceding the current, planned, intended imageacquisition. In some embodiments, the 3D representation of the surface,volume or combination thereof can comprise model data, e.g. a generic 3Dmodel of patient anatomy, an avatar of patient anatomy, etc. In someembodiments, the 3D representation can comprise a 2D outline, 3Doutline, a mesh, a group of surface points, or a combination thereof atleast in part derived from or based on the information about thegeometry of the one or more components of the imaging system,information about the position, orientation and/or geometry of the imageacquisition, information about the one or more image acquisitionparameters, or a combination thereof. The shape of the 3D representationof the surface, volume or combination thereof, optionally displayed, bya head mounted display or another augmented reality display device canchange responsive to changes in a geometry of one or more components ofthe imaging system (e.g. a collimator), changes in a position,orientation and/or geometry of the image acquisition (e.g. a change intube location, detector location, image intensifier location, field ofview, collimation), and/or changes in one or more image acquisitionparameters.

In some embodiments, the imaging system or one or more of its componentscan be moved, e.g. by a user, optionally assisted by one or more motors,controllers, drives, hydraulic and/or electric system, and/or roboticcomponents and/or arms, and the augmented view or 3D stereoscopic view,in some embodiments optionally superimposed onto the patient, can beconfigured by the system and/or one or more computer processor to movein relation with the tracked one or more components of the imagingsystem. When the imaging system or one or more of its components aremoved or when one or more parameters determining the location and/ororientation of the image acquisition are changed, the system can beconfigured to adjusting the position, orientation, position andorientation of the augmented view or 3D stereoscopic view in response tothe movement of the tracked imaging system, the movement of the one ormore of its components and/or the changes in the one or more parametersdetermining the location and/or orientation of the image acquisition. Insome embodiments, the adjusting or adjustment is configured to maintainthe augmented view at the defined position and orientation relative tothe one or more components of the imaging system. In some embodiments,the augmented view at the defined position relative to the one or morecomponents of the imaging system moves in relation with the tracked oneor more components of the imaging system, wherein the moving facilitatessuperimposing or aligning the 3D representation with a target anatomicstructure of the patient. In some embodiments, the imaging system can beconfigured to acquire 2D, 3D, or 2D and 3D imaging data of a patient,and wherein the 2D, 3D, or 2D and 3D imaging data of the patient areacquired within the 3D representation of the surface, volume orcombination thereof. In some embodiments, the step of generating theaugmented view is before the step of acquiring 2D, 3D, or 2D and 3Dimaging data of the patient, or wherein the step of displaying theaugmented view is before the step of acquiring 2D, 3D, or 2D and 3Dimaging data of the patient. In some embodiments, at least one computerprocessor can be configured to generate a 3D representation of asurface, a volume or combination thereof for display by an HMD or anaugmented reality device, wherein the 3D representation of the surface,volume or combination thereof can comprise information about a geometryof the one or more components of the imaging system, information about aposition, orientation and/or geometry of the image acquisition,information about one or more image acquisition parameters, or acombination thereof, wherein the 3D representation of the surface,volume or combination thereof can, for example, not contain imaging datafrom a patient. A user can move the 3D representation of the surface orvolume superimposed onto the patient, for example by interaction with avirtual user interface displayed by the HMD or an augmented reality ormixed reality display device. The movement of the 3D representation can,for example, be intended to move the 3D representation into a desiredanatomic area or volume (e.g. of a target anatomic area) to be includedand/or for inclusion in an imaging data acquisition. The system can thenoptionally be configured to move the imaging system, one or morecomponents of the imaging system, a patient table used in conjunctionwith the imaging system, or any combination thereof, for example usingone or more motors, controllers, drives, hydraulic and/or electricsystem, and/or robotic components and/or arms, responsive to themovement of the 3D representation. In this fashion, a subsequent imagingdata acquisition can be obtained in the desired area or volume of thesubsequent imaging data acquisition.

In some embodiments, the step of acquiring the 2D or 3D or 2D and 3Dimages can be after the step of generating the augmented view or 3Dstereoscopic view. In some embodiments, the step of acquiring the 2D or3D or 2D and 3D images can be after the step of displaying the augmentedview or 3D stereoscopic view.

In some embodiments, at least one computer processor can be configuredto generate a 3D representation of the surface, volume or combinationthereof before acquisition of 2D, 3D, or 2D and 3D imaging data of thepatient, or at least one computer processor can be configured to displaythe 3D representation of the surface, volume or combination thereofbefore acquisition of 2D, 3D, or 2D and 3D imaging data of the patient.

In some embodiments, a system of preparing an imaging data acquisitionassociated with a patient can comprise at least one computer processor,a head mounted display, e.g. a video see-through head mounted display oran optical see-through head mounted display, an augmented realitydisplay device, e.g. a tablet or smart phone, an imaging system, whereinthe at least one computer processor can be configured to obtainreal-time tracking information of one or more components of the imagingsystem (and optionally a patient table, the HMD, the augmented realitydisplay device, an anatomic structure of a patient, a robot), whereinthe at least one computer processor can be configured to generate a 3Drepresentation of a surface, a volume or combination thereof, whereinthe 3D representation of the surface, volume or combination thereof cancomprise information about a geometry of the one or more components ofthe imaging system, information about a position, orientation and/orgeometry of the image acquisition, information about one or more imageacquisition parameters, or a combination thereof, wherein the 3Drepresentation of the surface, volume or combination thereof can, forexample, not contain imaging data from a patient, or can contain imagingdata from a prior imaging data acquisition but not the current, planned,intended image acquisition, of the patient, wherein the at least onecomputer processor can be configured to generate an augmented view or 3Dstereoscopic view, the augmented view or 3D stereoscopic view comprisingthe 3D representation of the surface, volume or combination thereof,wherein the at least one computer processor can be configured todisplay, by the head mounted display or augmented reality device, theaugmented view or 3D stereoscopic view, e.g. superimposed onto thepatient or combined with a video feed of the patient or the imagingsystem (if an augmented reality display device such as a table or smartphone is used), wherein the position and/or orientation of the augmentedview or 3D stereoscopic view (including, optionally, an intendedposition and/or orientation) can be determined based on the real timetracking information of the one or more components of the imagingsystem, and wherein the imaging system can be configured to acquire 2D,3D, or 2D and 3D imaging data of the patient within the or at thelocation of the 3D representation of the surface, volume or combinationthereof. In some embodiments, the imaging system can also acquireimaging studies with more than 3 dimensions, four example whensequential scanning is performed with time being the 4^(th) dimension.Representative examples comprise, for example, ultrasound flow studies,Doppler ultrasound, angiography including vascular run-offs, spiral CTimaging after contrast bolus for vascular imaging (e.g. through orduring a cardiac cycle) or MR angiography studies or imaging combinationstudies, e.g. SPECT-CT or PET-MRI.

In some embodiments, the at least one computer processor can beconfigured to generate the 3D representation of the surface, volume orcombination thereof before acquisition of the 2D, 3D, or 2D and 3Dimaging data of the patient and/or in some embodiments, the at least onecomputer processor can be configured to display the 3D representation ofthe surface, volume or combination thereof before acquisition of the 2D,3D, or 2D and 3D imaging data of the patient.

In some embodiments, at least one computer processor can be configuredto generate a 3D representation of a surface, volume or combinationthereof, which can comprise information about a limit, an edge, amargin, a boundary, a circumference, a perimeter, or a combinationthereof of a planned or intended or upcoming imaging data acquisition,e.g. a 2D, 3D, 4D, 5D or combination thereof imaging acquisition. Thelimit, edge, margin, boundary, circumference, perimeter, or acombination thereof of a planned or intended or upcoming imaging dataacquisition can not contain or comprise any imaging data of the patientor can comprise imaging data of the patient from a prior imaging dataacquisition, but not from the planned or intended or upcoming imagingdata acquisition. The limit, edge, margin, boundary, circumference,perimeter, or a combination thereof of a planned or intended or upcomingimaging data acquisition can comprise information about a geometry ofthe one or more components of the imaging system, information about aposition, orientation and/or geometry of the image acquisition,information about one or more image acquisition parameters, or acombination thereof. The limit, edge, margin, boundary, circumference,perimeter, or a combination thereof of a planned or intended or upcomingimaging data acquisition can be based on or can be derived frominformation about a geometry of the one or more components of theimaging system, information about a position, orientation and/orgeometry of the image acquisition, information about one or more imageacquisition parameters, or a combination thereof.

The location, position, and/or orientation of the limit, edge, margin,boundary, circumference, perimeter, or a combination thereof, thelocation, position, and/or orientation of the surface or volume of the3D representation of a planned or intended or upcoming imaging dataacquisition can be based on the tracked location, position, orientation,and/or coordinates of the imaging system, and/or the tracked location,position, orientation, and/or coordinates of one or more components ofthe imaging system.

In some embodiments, the at least one computer processor can beconfigured to generate the surface, volume or combination thereof atleast in part from information about a geometry of the imaging system,information about a position, orientation and/or geometry of one or morecomponents of the imaging system, information about a geometry of theimage acquisition, information about one or more image acquisitionparameters, or a combination thereof. In some embodiments, the at leastone computer processor can be configured to determine a desired locationand orientation of the augmented view or 3D stereoscopic view associatedwith the imaging system, wherein the at least one computer processor canbe configured to generate the 3D representation of the surface, volumeor combination thereof before acquisition of the 2D, 3D, or 2D and 3Dimaging data of the patient, or wherein the at least one computerprocessor can be configured to display the 3D representation of thesurface, volume or combination thereof before acquisition of the 2D, 3D,or 2D and 3D imaging data of the patient.

In some embodiments, the system can be configured to acquire the 2D, 3D,or 2D and 3D imaging data from the patient by the imaging system at thelocation and orientation of the 3D representation of the surface, volumeor combination thereof and/or at the location and orientation of thestereoscopic view or augmented view, displayed by an HMD or an augmentedreality display device such as a tablet or smart phone, of the 3Drepresentation of the surface, volume or combination thereof.

In some embodiments, the at least one computer processor can beconfigured to project the augmented view or 3D stereoscopic view of the3D representation of the surface, volume or combination thereof (basedat least in part on information about a geometry of the imaging system,information about a position, orientation and/or a geometry of one ormore components of the imaging system, information about a geometry ofthe image acquisition, information about one or more image acquisitionparameters, or a combination thereof) at the coordinates of the plannedimaging data acquisition of the patient, which can be a 2D, 3D, 4D, 5Dor more imaging data acquisition (accounting, for example, also fortemporal elements, subtraction imaging etc.). In some embodiments, thelocation of the imaging data acquisition, e.g. a 2D, 3D, 4D orcombination thereof imaging data acquisition, can comprise one or moreanatomic structures of the patient. The anatomic structures canoptionally be tracked using any of the techniques described in thespecification or known in the art.

In some embodiments, the system can be configured to facilitatedetermining a desired position and orientation of the augmented view,wherein the desired position and orientation comprises a target anatomicstructure of the patient. In some embodiments, the at least one computerprocessor can be configured to adjust the position, orientation, orposition and orientation of the augmented view or 3D stereoscopic viewin response to movement of one or more tracked components of the imagingsystem and/or movement of the patient table. The adjustment can beconfigured to maintain the augmented view at the defined position andorientation relative to the one or more components of the imagingsystem.

In some embodiments, the information (for example used for generatingthe representation of the surface or volume of the planned imaging dataacquisition) about the geometry of the imaging system, information aboutthe position, orientation and/or geometry of one or more imaging systemcomponents, information about the geometry of the image acquisition,information about one or more image acquisition parameter, or acombination thereof can comprise information about one or more imagingsystem components, a geometric relationship between one or more imagingsystem components, a collimator, a grid, an image intensifier, adetector resolution, an x-ray source, an x-ray tube setting, a kVpsetting, an mA setting, an mAs setting, a collimation, a tube—detectordistance, a tube—patient distance, patient—detector distance, apatient—image intensifier distance, a table height relative to a tube, adetector, a table position relative to a tube, a detector, orcombination thereof, a patient position, a C-arm position, orientation,or combination thereof, a gantry position, orientation or combinationthereof, a grid height, a grid width, a grid ratio, a field of view, acenter of a field of view, a margin or periphery of a field of view, amatrix, a pixel size, a voxel size, an image size, an image volume, animaging plane, an image dimension in x, y, z and/or oblique direction,an image location, an image volume location, a scan coverage, a pitch,an in-plane resolution, a slice thickness, an increment, a detectorconfiguration, a detector resolution, a detector density, a tubecurrent, a tube potential, a reconstruction algorithm, a scan range, ascan boundary a scan limit, a rotational center of a scan acquisition, arotational axis of a scan acquisition, a reconstructed slice thickness,a segmentation algorithm, a window, a level, a brightness, a contrast, adisplay resolution, or any other setting parameter tabulated in Table 1,or any combination thereof. The geometric relationship between one ormore imaging system components can, for example, comprise atube—detector distance, a tube—image intensifier distance, a rotationalaxis (e.g. in a spiral CT or cone beam CT or 3D C-arm), a center ofrotation, an MRI coil dimension, a sensitivity profile (e.g. for an MRIprofile) in relationship to one or more dimensions/geometries etc. Theinformation about a collimator can comprise, for example, a gridspacing, a grid depth, a collimator dimension and any changes thereto,e.g. induced by movement of one or more motors, controllers and ordrives.

In some embodiments, an imaging system can comprise an x-ray system, afluoroscopy system, a C-arm, a 3D C-arm, a digital tomosynthesis imagingsystem, an angiography system, a bi-planar angiography system, a 3Dangiography system, a CT scanner, an MRI scanner, a PET scanner, a SPECTscanner, a nuclear scintigraphy system, a 2D ultrasound imaging system,a 3D ultrasound imaging system, or a combination thereof.

In some embodiments, at least one computer processor can be configuredto obtain real-time tracking information of a head mounted display, anaugmented reality display device, an anatomic structure of the patient,a patient table used with the imaging system, an imaging system, one ormore components of the imaging system, a surgical instrument, a surgicaltool, an implant, a surgical robot, a robot integrated with or part ofthe imaging system, or any combination thereof. Any of the trackingtechniques described in the specification using extrinsic and intrinsictracking can be used. For example, a surgical navigation system, camera,3D scanner can be used. Inside out or outside in tracking can be used.Intrinsic information can be used for tracking as described throughoutthe specification.

In some embodiments, the system can comprise a camera or scannerconfigured to acquire the real-time tracking information of the headmounted display, the anatomic structure of the patient, the patienttable used with the imaging system, the imaging system, the one or morecomponents of the imaging system, a surgical instrument, a surgicaltool, an implant or a combination thereof. In some embodiments, a cameraor scanner comprise at least one of a navigation system, a 3D scanner, aLIDAR system, a depth sensor, an IMU or a combination thereof. In someembodiments, tracking information can comprise coordinate information ofa head mounted display, an augmented or mixed reality display device, ananatomic structure of the patient, a patient table used with the imagingsystem, an imaging system, one or more components of the imaging system,or a combination thereof. In some embodiments, tracking information cancomprise location information of a head mounted display, an augmented ormixed reality display device, an anatomic structure of the patient, apatient table used with the imaging system, an imaging system, one ormore components of the imaging system components, or a combinationthereof. In some embodiments, a camera or scanner can comprise a laserscanner, time-of-flight 3D scanner, structured-light 3D scanner,hand-held laser scanner, a time-of-flight camera or a combinationthereof.

In some embodiments, a system can be configured to obtain real timetracking information of an imaging system using intrinsic informationfrom the imaging system. Intrinsic information can comprise pose data,sensor data, camera data, 3D scanner data, controller data, drive data,actuator data, end effector data, data from one or more potentiometers,data from one or more video systems, data from one or more LIDARsystems, data from one or more depth sensors, data from one or moreinertial measurement units, data from one or more accelerometers, datafrom one or more magnetometers, data from one or more gyroscopes, datafrom one or more force sensors, data from one or more pressure sensors,data from one or more position sensors, data from one or moreorientation sensors, data from one or more motion sensors, positionand/or orientation data from step motors, position and/or orientationdata from electric motors, position and/or orientation data fromhydraulic motors, position and/or orientation data from electric and/ormechanical actuators, position and/or orientation data from drives,position and/or orientation data from robotic controllers, positionand/or orientation data from one or more robotic computer processors, ora combination thereof.

In some embodiments, an imaging system, e.g. an x-ray system,fluoroscopy system, C-arm, 3D C-arm, cone beam CT, CT scanner, SPECT CTscanner, PET CT scanner can be configured to generate an x-ray beam. Insome embodiments, an x-ray beam of an imaging system can be cone shapedor cylindrical, pencil shaped, or fan shaped. In some embodiments, anx-ray beam of an imaging system can originate from one or more pointsources. In some embodiments, an x-ray beam of can imaging system can becollimated. The collimation can be adjustable. The adjustment of thecollimation can be performed using a graphical user interface, e.g. avirtual user interface displayed by an HMD. The collimation candetermine the shape, limit, margin, perimeter of a 3D representation ofa surface or volume of an image acquisition. The 3D representation canbe displayed in stereoscopic form by an HMD, superimposed onto a patientlying on a patient table in the imaging system.

In some embodiments, the system can comprise a user interface. In someembodiments, a user interface can comprise a virtual user interface,wherein the virtual interface can comprise at least one virtual object.In some embodiments, a virtual object, e.g. displayed as part of avirtual interface by an HMD or other augmented reality display device,can comprise one or more virtual button, virtual field, virtual cursor,virtual pointer, virtual slider, virtual trackball, virtual node,virtual numeric display, virtual touchpad, virtual keyboard, or acombination thereof. In some embodiments, a virtual user interface cancomprise a gesture recognition, gaze recognition, gaze lock or acombination thereof. Other interfaces can comprise a voice recognition,eye tracking, hand tracking, pointer tracking, instrument tracking, tooltracking, or a combination thereof, which can optionally also be used to“touch” and/or activate one or more virtual objects displayed by anvirtual interface. In some embodiments, at least one computer processorcan be configured to generate a command based at least in part on atleast one interaction of a user with at least one virtual objectdisplayed by an HMD or other augmented reality display device in avirtual user interface. In some embodiments, a command can be configuredto move, tilt, or rotate one or more components of an imaging system,one or more components of a patient table or a combination thereof. Insome embodiments, a command can be configured to activate, operate,de-activate or a combination thereof a motor, an actuator, a drive, acontroller, a hydraulic system, a switch, an x-ray tube, an imageintensifier, or an imaging system or an ultrasound transducer, anultrasound receiver, a robot (e.g. a surgical robot) or a combinationthereof.

In some embodiments, a command, e.g. from a virtual interface, can beconfigured to move one or more components of a robot and/or one or moreend effectors, to move or modify a geometry of am imaging system, apatient table, a geometric relationship between one or more imagingsystem components, a collimator, a grid, an image intensifier, adetector resolution, a setting of the imaging system, a parameter of theimaging system, a parameter of the imaging data acquisition, a displayparameter, an x-ray source setting, an x-ray tube setting, a kVpsetting, an mA setting, an mAs setting, a collimation, a tube—detectordistance, a tube—patient distance, patient—detector distance, apatient—image intensifier distance, a center of rotation, e.g. of animage acquisition, a center of rotation of a rotational movement of atube and/or detector system, a center of rotation of a rotationalmovement of a C-arm, a center of rotation of a rotational movement of acone beam CT scanner, a center of a spiral acquisition of a spiral CTscanner, a table height relative to a tube, a detector, a table positionrelative to a tube, a detector, a patient position, a C-arm position,orientation, or combination thereof, a gantry position, orientation orcombination thereof, a grid height, a grid width, a grid ratio, a fieldof view, a matrix, a pixel size, a voxel size, an image size, an imagevolume, an imaging plane, an image dimension in x, y, z and/or obliquedirection, an image location, an image volume location, a scan coverage,a pitch, an in-plane resolution, a slice thickness, an increment, adetector configuration, a detector resolution, a detector density, atube current, a tube potential, a reconstruction algorithm, a scan rangea scan boundary, a scan limit, a reconstructed slice thickness, asegmentation algorithm, a window a level, a brightness, a contrast, adisplay resolution, or a combination thereof.

In some embodiments, a command can be configured to set and/or modifyone or more image acquisition parameters of an imaging system, e.g. anx-ray system, C-arm, 3D C-arm, cone beam CT system, CT scanner, spiralCT scanner, MRI scanner, ultrasound system, radionuclide imaging device,SPECT scanner, PET system, cardiac imaging system, angiography system,or any combination thereof.

In some embodiments, a command can be configured to set, move, and/ormodify a position, orientation, size, area, volume, or combinationthereof of am imaging data acquisition, e.g. a 2D, 3D, 4D, 5D imagingdata acquisition or any combination thereof. In some embodiments, acommand can be configured to set, move, and/or modify one or morecoordinates of the 3D representation, e.g. of a surface or volumerepresenting the limit, perimeter, outer margin of an upcoming, planned,intended image acquisition, e.g. for stereoscopic display by an HMD oraugmented display, e.g. of a video feed from an imaging system, apatient etc., by another augmented reality display device. In someembodiments, a command can be configured to set, move, and/or modify oneor more coordinates of an upcoming, planned, intended imaging dataacquisition, e.g. a 2D, 3D, 4D, 5D imaging data acquisition or anycombination thereof. In some embodiments, a command can be configured toset, move and/or modify a dimension, a size, an area, a volume or acombination thereof of a 3D representation, e.g. of a surface or volumerepresenting the limit, perimeter, outer margin of an upcoming, planned,intended image acquisition, e.g. for stereoscopic display by an HMD.

In some embodiments, a command can be configured to set, move and/ormodify a dimension, a size, an area, a volume or a combination thereofof an upcoming, planned, intended imaging data acquisition, e.g. a 2D,3D, 4D, 5D imaging data acquisition or any combination thereof. In anyembodiments throughout the specification, a command can be configured toactivate, operate, de-activate or a combination thereof a sensor, acamera, a video system, a 3D scanner, a Lidar system, a navigationsystem, a potentiometer, an electronic circuit, a computer chip, apiezoelectric system, a piezoelectric mechanism, a piezoelectric lock orrelease system, a controller, a drive, a motor, a hydraulic system, anactuator, a functional unit or a combination thereof of an imagingsystem, an imaging system component, a patient table or a combinationthereof.

In any embodiments throughout the specification, a command can beconfigured to activate, operate, de-activate or a combination thereof asensor, a camera, a video system, a 3D scanner, a Lidar system, anavigation system, a potentiometer, an electronic circuit, a computerchip a piezoelectric system, a piezoelectric mechanism, a piezoelectriclock or release system, a controller, a drive, a motor, a hydraulicsystem, an actuator, or a combination thereof of a surgical robot, acomponent of a surgical robot, a functional unit of a surgical robot, apatient table used in conjunction with a surgical robot or a combinationthereof.

In any embodiments throughout the specification, a sensor can comprise adepth sensor, inertial measurement unit, accelerometer, magnetometer,gyroscope, force sensor, pressure sensor, position sensor, orientationsensor, motion sensor, or a combination thereof.

In some embodiments, one or more components of an imaging system can beattached to, integrated with, or part a robot. A robot can be configuredto move one or more components of the imaging system.

In some embodiments, a virtual user interface can be configured togenerate an event message triggered by a user interaction, e.g. acollision detection. The system can comprise an event handler configuredto process the event message. An event handler can be configured togenerate a command. In some embodiments, a computing system can beconfigured to generate a command, wherein the command can be triggeredby a virtual user interface.

In some embodiments, a system can be configured to determine/identify adesired location/orientation of an augmented view or 3D stereoscopicview, optionally associated with the imaging system, to acquire imagingdata, e.g. 2D, 3D, 4D or any combination thereof, at a desiredlocation/orientation.

In some embodiments, a system can be configured to determine a desiredlocation of an augmented view associated with the imaging system toacquire 2D, 3D, or 2D and 3D imaging data at the desired location. Insome embodiments, an augmented reality display device can be a headmounted display, and wherein the augmented view can comprise a 2D, 3Dand/or a 3D stereoscopic view. In some embodiments, at least onecomputer processor can be configured to project the 3D stereoscopic viewat the coordinates of intended 2D, 3D or 2D and 3D imaging dataacquisition of the patient. In some embodiments, the location of the 2D,3D, or 2D and 3D imaging data acquisition comprises one or more targetanatomic structures of the patient.

FIGS. 6A-6B show non-limiting examples of a C-arm system, e.g. a 2D or3D C-arm, and applications of a virtual display by a head mounteddisplay (HMD) or other augmented reality display device, e.g. a tablet,iPad, smart phone, iPhone etc. One or more users 1600 can wear a headmounted display (HMD) 1610. The head mounted display or other augmentedreality display device can generate a virtual display of one or morevirtual objects within a field of view 1620 of the HMD 1610 or otheraugmented reality display device. A virtual display can comprise a 3Drepresentation 1630 of a surface or volume of an energy beam, e.g. anx-ray beam 1640. The surface of volume can describe an outer envelope,margin, perimeter, of the energy beam 1640. The 3D representation andthe stereoscopic display by the HMD or other augmented reality displaydevice can be generated prior to activating/turning on the energy beam.The 3D representation 1630 of the surface or volume can be basedon/derived from a geometry of the one or more components of the imagingsystem, information about a geometry of the image acquisition,information about one or more image acquisition parameters, or acombination thereof. The 3D representation 1630 of the surface, volumeor combination thereof can, for example, not contain imaging data from apatient 1650, or can contain imaging data from a prior imaging dataacquisition but not the current, planned, intended, desired imageacquisition, of the patient 1650. Also shown are an x-ray tube 1660 andan image intensifier 1670 of a C-arm system. The position, orientation,position and orientation of the 3D representation 1630 can be defined inrelationship to one or more components of the imaging system, thepatient table, or a combination thereof. The position, orientation,position and orientation of the 3D representation 1630 can be fixed inrelationship to one or more components of the imaging system; the systemcan be configured that the 3D representation moves with movement of theone or more components of the imaging system, and/or any changes in theposition, orientation, and/or geometry of the imaging system, anychanges in the position, orientation, and/or geometry of one or moreimaging system components (including, for example, also a collimator),any changes in the position, orientation, and/or geometry of the imageacquisition, any changes in one or more image acquisition parameters ora combination thereof.

FIG. 6C-6E show non-limiting examples of a 3D C-arm system andapplications of a virtual display by a head mounted display (HMD) orother augmented reality display device, e.g. with augmented reality orstereoscopic display of a 3D representation 1700 of a surface or volumerepresenting, for example, an outer envelope or perimeter or limit of anintended or planned 3D imaging data acquisition, prior to the actual 3Dimaging data acquisition. The view 1620 through an HMD or otheraugmented reality device shows a virtual display, by the HMD or otheraugmented reality display device, of a 3D representation 1700 of asurface or volume of a 3D imaging data volume acquisition intended for apatient 1650, displayed by the HMD or other augmented reality displaydevice prior to the actual 3D imaging data volume acquisition. Theimaging data volume acquisition with the 3D C-arm 1680, in thisnon-limiting example, (or with any 3D or 4D imaging system) intended inthis patient 1650 can be a volumetric, cylinder shaped acquisition, inthis example. Any other 3D shapes of image acquisitions are possible,depending on the imaging system used, its components and its uniquegeometries. The 3D representation 1700 of the surface or volume of the3D imaging data acquisition can be based on/derived from a position,orientation and/or geometry of the one or more components of the imagingsystem, information about a position, orientation and/or geometry of theimage acquisition, information about one or more image acquisitionparameters, or a combination thereof. The 3D representation 1700 of thesurface or volume of the intended 3D imaging data acquisition can bedisplayed, for example in an overlay fashion, superimposed onto thepatient 1650 and/or a target organ or target anatomic structure, such asa portion of a spine 1710 in this non-limiting example. The imagingsystem, in this non-limiting example a 3D C-arm 1680, and/or one or moreof its components, and/or the patient table 1690 can be moved, e.g. by auser, optionally assisted by one or more motors, controllers, drives,hydraulic and/or electric system, and/or robotic components and/or arms.Optionally, the imaging system, in this non-limiting example a 3D C-arm1680, and/or one or more of its components, and/or the patient table1690 can be tracked, e.g. using any of the techniques describedthroughout the specification. In addition, the patient 1650 and/or thetarget organ or target anatomic structure, such as a portion of a spine1710 can also optionally be tracked, e.g. using any of the techniquesdescribed throughout the specification. The system, including one ormore of its computing systems and/or computer processors, can beconfigured so that the 3D stereoscopic view or augmented view of the 3Drepresentation 1700 of the surface or volume of the envelope, perimeter,limits, margin, or combination thereof of the intended or desiredimaging data acquisition, for example, superimposed, by the HMD or otheraugmented reality display device, onto the patient (prior to the actualimaging data acquisition) can move in relation with the tracked imagingsystem 1680, tracked one or more components of the imaging system 16601670, tracked patient table 1690, and/or, optionally, the trackedanatomic target structure, such as a spine 1710 (e.g. optionally withone or more attached markers or fiducials or fiducial arrays, notshown). When the imaging system 1680 or one or more of its components1660 1670 are moved, or when the patient table 1690 is moved, or whenone or more parameters determining and/or influencing the locationand/or orientation of the image acquisition are changed, or anycombination thereof, the system can be configured to adjusting theposition, orientation, position and orientation of the 3D stereoscopicview or augmented view of the 3D representation 1700 in response to themovement of the tracked imaging system 1680, the movement of the one ormore of its components 1660 1670, the movement of the tracked patienttable 1690, and/or the changes in the one or more parameters determiningand/or influencing the location and/or orientation of the imageacquisition.

FIGS. 6C-6E show how the 3D representation 1700 of the surface or volumeof the envelope, perimeter, limits, margin, or combination thereof ofthe intended or desired imaging data acquisition displayed by the HMD orother augmented reality display device at a defined position andorientation relative to the one or more components of the imaging system(optionally superimposed onto the patient) prior to the actual imagingdata acquisition moves in relationship to a target anatomic structure,such as a spine 1710 with the movement of the tracked imaging system1680, the movement of the one or more of its components 1660 1670,optionally the movement of the tracked patient table 1690, and/or thechanges in the one or more parameters determining and/or influencing thelocation and/or orientation of the image acquisition.

In FIG. 6C, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed, by the HMD orother augmented reality device, in this example onto the patient 1650prior to the actual imaging data acquisition is too far anteriorrelative to a target anatomic structure, such as a spine 1710. This canoccur, for example, when the imaging system 1680 or one or more of itscomponents are too low in position relative to the patient 1650 or thetarget anatomic structure, such as a spine 1710, or when the patienttable 1690 is too high relative to the imaging system 1680. An explodedview 1720, shows what the images will look like after the image dataacquisition. Only a small anterior portion of the spine 1730 will beincluded in the actual image data volume acquisition and the resultantimages.

In FIG. 6D, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition in this examplesuperimposed by the HMD or other augmented reality device onto thepatient 1650 prior to the actual imaging data acquisition is too farlateral, off to the side, relative to a target anatomic structure, suchas a spine 1710. This can occur, for example, when the imaging system1680 or one or more of its components are too far lateral or mal-alignedin position relative to the patient 1650 or the target anatomicstructure, such as a spine 1710, or when the patient table 1690 is toofar lateral, off to the side relative to the imaging system 1680, e.g.the center of the imaging system, the center of a rotation of a C-arm,the center of a spin of a cone beam CT, and/or the center of a spiral,e.g. with a CT scan, or when the patient 1650 is not correctlypositioned on the patient table 1690, e.g. not in the center of thepatient table, off to the side. An exploded view 1720, shows what theimages will look like after the actual image data acquisition. Only asmall lateral portion of the spine 1730 will be included in the imagedata volume acquisition and the resultant images.

In FIG. 6E, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed in thisexample by the HMD or other augmented reality display device onto thepatient 1650 prior to the actual imaging data acquisition is correctlypositioned relative to a target anatomic structure, such as a spine1710. This can occur, for example, when the imaging system 1680 or oneor more of its components are correctly positioned, e.g. centeredrelative to the patient 1650 or the target anatomic structure, such as aspine 1710, and/or when the patient table 1690 is correctly positionedrelative to the imaging system 1680, and/or when the patient 1650 iscorrectly positioned on the patient table 1690, e.g. in the center ofthe patient table, not off to the side. An exploded view 1720, showswhat the images will look like after actual the image data acquisition.The entire target portion of the spine 1730 will be included in theimage data volume acquisition and the resultant images.

FIGS. 7A-7C show non-limiting examples of a radiography (e.g. 2D, 3D)1750, angiography (e.g. 2D, 3D, 4D) 1750 or other x-ray based imagingsystem 1750, and applications of a virtual display by a head mounteddisplay (HMD) or other augmented reality display device. One or moreusers 1600 can wear a head mounted display (HMD) 1610 or use anothertype of augmented reality display device. The head mounted display orother augmented reality display device can generate a virtual display ofone or more virtual objects within a field of view 1620 of the HMD 1610or other augmented reality display device. A virtual display cancomprise a 3D representation 1630 of a surface or volume of an energybeam, e.g. an x-ray beam. The surface of volume can describe an outerenvelope, margin, perimeter, of the energy beam. The 3D representationand the stereoscopic display by the HMD or other augmented realitydisplay device can be generated prior to activating/turning on theenergy beam, e.g. x-ray beam. The 3D representation 1630 of the surfaceor volume can be based on/derived from a position, orientation, and/orgeometry of the one or more components of the imaging system,information about a position, orientation and/or geometry of the imageacquisition, information about one or more image acquisition parameters,or a combination thereof. The 3D representation 1630 of the surface,volume or combination thereof can, for example, not contain imaging datafrom a patient 1650, or can contain imaging data from a prior imagingdata acquisition but not the current, planned, intended, desired imageacquisition, of the patient 1650. Also shown is an x-ray tube housing1665. In FIG. 7B, the x-ray beam and corresponding 3D representation1630 are cone shaped with a round base. In FIG. 7C, the x-ray beam andcorresponding 3D representation 1630 have a rectangular or quadraticperimeter with a rectangular or quadratic base.

FIG. 7D-7E show non-limiting examples of a radiography (e.g. 3D) 1750,angiography (e.g. 3D, 4D) 1750 or other x-ray based imaging system 1750,and applications of a virtual display by a head mounted display (HMD),e.g. with stereoscopic display of a 3D representation 1700 of a surfaceor volume representing, for example, an outer envelope or perimeter orlimit of an intended or planned 3D (or, optionally 4D) imaging dataacquisition, prior to the actual 3D (or, optionally 4D) imaging dataacquisition. The view 1620 through the HMD or other augmented realitydevice shows a virtual display, by the HMD, of a 3D representation 1700of a surface or volume of a 3D imaging data volume acquisition intendedfor a patient 1650, displayed by the HMD or other augmented realitydevice prior to the actual 3D imaging data volume acquisition. Theimaging data volume acquisition with the radiography (e.g. 3D) 1750,angiography (e.g. 3D, 4D) 1750 or other x-ray based imaging system 1750,in this non-limiting example, (or with any 3D or 4D imaging system)intended in this patient 1650 can be a volumetric, cylinder shapedacquisition, in this example. Any other 3D shapes of image acquisitionsare possible, depending on the imaging system used, its componentsand/or its unique geometries. The 3D representation 1700 of the surfaceor volume of the 3D imaging data (or, optionally, 4D) acquisition can bebased on/derived from a position, orientation and/or geometry of the oneor more components of the imaging system, information about a position,orientation and/or geometry of the image acquisition, information aboutone or more image acquisition parameters, or a combination thereof. The3D representation 1700 of the surface or volume of the intended 3Dimaging data acquisition can be displayed, for example in an overlayfashion, superimposed onto the patient 1650 and/or a target organ ortarget anatomic structure, such as a portion of a spine 1710 by an HMDor other augmented reality device, e.g. a table, iPad, smart phone etc.,in this non-limiting example.

The imaging system, in this non-limiting example a radiography (e.g. 3D)1750, angiography (e.g. 3D, 4D) 1750 or other x-ray based imaging system1750, and/or one or more of its components, and/or the patient table1690 can be moved, e.g. by a user, optionally assisted by one or moremotors, controllers, drives, hydraulic and/or electric system, and/orrobotic components and/or arms. Optionally, the imaging system, in thisnon-limiting example a radiography (e.g. 3D) 1750, angiography (e.g. 3D,4D) 1750 or other x-ray based imaging system 1750, and/or optionally oneor more of its components, and/or optionally the patient table 1690 canbe tracked, e.g. using any of the techniques described throughout thespecification. In addition, the patient 1650 and/or the target organ ortarget anatomic structure, such as a portion of a spine 1710, canoptionally also be tracked, e.g. using any of the techniques describedthroughout the specification. The system, including one or more of itscomputing systems and/or computer processors, can be configured so thatthe 3D stereoscopic view of the 3D representation 1700 of the surface orvolume of the envelope, perimeter, limits, margin, or combinationthereof of the intended or desired imaging data acquisition displayed bythe HMD or other augmented reality device at a defined position,orientation, coordinates or combination thereof in relationship to theone or more components of the imaging system (and, as in this example,onto the patient) prior to the actual imaging data acquisition can movein relation with the tracked imaging system 1750, tracked one or morecomponents of the imaging system, optionally tracked patient table 1690,and/or the (optionally tracked) anatomic target structure, such as aspine 1710 (e.g optionally with one or more attached markers orfiducials or fiducial arrays, not shown). When the imaging system 1750or one or more of its components are moved, or, optionally, when thepatient table 1690 is moved, or when one or more parameters determiningand/or influencing the location and/or orientation of the imageacquisition are changed, or any combination thereof, the system can beconfigured to adjusting the position, orientation, position andorientation of the 3D stereoscopic view or augmented view of the 3Drepresentation 1700 in response to the movement of the tracked imagingsystem 1750, the movement of the one or more of its components, themovement of the tracked patient table 1690, and/or the changes in theone or more parameters determining and/or influencing the locationand/or orientation of the image acquisition.

FIGS. 7D-7E show how the 3D representation 1700 of the surface or volumeof the envelope, perimeter, limits, margin, or combination thereof ofthe intended or desired imaging data acquisition displayed, by the HMDor other augmented reality device, in a defined position, orientationand/or coordinates relative to the one or more components of the imagingmoves in relationship to a target anatomic structure, such as a spine1710, with the movement of the tracked imaging system 1750, the movementof the one or more of its components, the movement of the trackedpatient table 1690, and/or the changes in the one or more parametersdetermining and/or influencing the location and/or orientation of theimage acquisition.

In FIG. 7D, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed, by the HMD orother augmented reality device, onto the patient 1650 prior to theactual imaging data acquisition is too far anterior relative to a targetanatomic structure, such as a spine 1710. This can occur, for example,when the imaging system 1750 or one or more of its components are toolow in position relative to the patient 1650 or the target anatomicstructure, such as a spine 1710, or when the patient table 1690 is toohigh relative to the imaging system 1750. An exploded view 1720, showswhat the images will look like after the image data acquisition. Only asmall anterior portion of the spine 1730 will be included in the actualimage data volume acquisition and the resultant images.

In FIG. 7E, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed, by the HMD orother augmented reality device, onto the patient 1650 prior to theactual imaging data acquisition is correctly positioned relative to atarget anatomic structure, such as a spine 1710. This can occur, forexample, when the imaging system 1750 or one or more of its componentsare correctly positioned, e.g. centered relative to the patient 1650 orthe target anatomic structure, such as a spine 1710, and/or when thepatient table 1690 is correctly positioned relative to the imagingsystem 1750, and/or when the patient 1650 is correctly positioned on thepatient table 1690, e.g. in the center of the patient table, not off tothe side. An exploded view 1720, shows what the images will look likeafter actual the image data acquisition. The entire target portion ofthe spine 1730 will be included in the image data volume acquisition andthe resultant images.

Thus, by moving one or more of the tracked imaging system 1750, trackedone or more components of the imaging system, optionally tracked patienttable 1690, and/or the patient 1650, and/or the anatomic targetstructure, such as a spine 1710 prior to the actual image dataacquisition and by optimizing the position, orientation, or position andorientation and/or coordinates of the 3D representation 1700, displayedby the HMD or other augmented reality display device, of a surface orvolume of the envelope, margin, perimeter and/or limits of the 3Dimaging data volume acquisition intended for a patient 1650, and/or thetarget anatomic target structure, such as a spine 1710, the actual imagedata acquisition can be optimized with regard to the coverage and/orinclusion of the target anatomic target structure, such as a spine 1710.This approach can obviate the need for repeat scout images and/or x-rayimages (e.g. AP, lateral) prior to the actual 3D volume acquisition.

FIGS. 8A-8B show non-limiting examples of a CT, cone beam CT, spiral CT,MRI system, SPECT system, PET system 1780 or a combination thereof, andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device. One or more users 1600 can wear a headmounted display (HMD) 1610 or carry another augmented reality device.The head mounted display or other augmented reality device can generatea virtual display of one or more virtual objects within a field of view1620 of the HMD 1610 or other augmented reality device. A virtualdisplay can comprise a 3D representation 1630 of a surface or volume ofan energy beam 1640, e.g. an x-ray beam. The surface of volume candescribe an outer envelope, margin, perimeter, of the energy beam 1640.The 3D representation and the stereoscopic display or augmented view bythe HMD or other augmented reality device can be generated prior toactivating/turning on the energy beam 1640, e.g. x-ray beam. The 3Drepresentation 1630 of the surface or volume can be based on/derivedfrom a position, orientation and/or geometry of the one or morecomponents of the imaging system (e.g. an imaging transmit and/orreceiver coil with an MRI system, one or more gradients, gradient coils,etc.), information about a position, orientation, and/or geometry of theimage acquisition, information about one or more image acquisitionparameters, or a combination thereof. The 3D representation 1630 of thesurface, volume or combination thereof can, for example, not containimaging data from a patient 1650, or can contain imaging data from aprior imaging data acquisition but not the current, planned, intended,desired image acquisition, of the patient 1650.

FIG. 8C-8E show non-limiting examples of a CT, cone beam CT, spiral CT,MRI system, SPECT system, PET system 1780 or a combination thereof, andapplications of a virtual display by a head mounted display (HMD) orother augmented reality device, e.g. with stereoscopic and/or augmenteddisplay of a 3D representation 1700 of a surface or volume representing,for example, an outer envelope or perimeter or limit of an intended orplanned 3D (or, optionally 4D [e.g. vascular flow or angiographic, forexample with spiral CT angiography or MRI angiography]) imaging dataacquisition, prior to the actual 3D (or, optionally 4D) imaging dataacquisition. The view 1620 through the HMD or other augmented realitydevice shows a virtual and/or augmented display, by the HMD or otheraugmented reality device, of a 3D representation 1700 of a surface orvolume of a 3D imaging data volume acquisition intended for a patient1650, displayed by the HMD prior to the actual 3D imaging data volumeacquisition. The imaging data volume acquisition with the CT, cone beamCT, spiral CT, MRI system, SPECT system, PET system 1780 or acombination thereof, in this non-limiting example, (or with any 3D or 4Dimaging system) intended in this patient 1650 can be a volumetric,cylinder shaped acquisition, in this example. Any other 3D shapes ofimage acquisitions are possible, depending on the imaging system used,its components and its unique geometries. The 3D representation 1700 ofthe surface or volume of the 3D imaging data (or, optionally, 4D)acquisition can be based on/derived from a position, orientation and/orgeometry of the one or more components of the imaging system,information about a position, orientation and/or geometry of the imageacquisition, information about one or more image acquisition parameters,or a combination thereof. The 3D representation 1700 of the surface orvolume of the intended 3D imaging data acquisition can be displayed bythe HMD or other augmented reality device in a defined position,orientation and/or at defined coordinates relative to one or morecomponents of the imaging system, for example in an overlay fashionsuperimposed onto a patient 1650 and/or a target organ or targetanatomic structure, such as a portion of a spine 1710 in thisnon-limiting example. The imaging system, in this non-limiting example aCT, cone beam CT, spiral CT, MRI system, SPECT system, PET system 1780or a combination thereof, and/or one or more of its components, and/orthe patient table 1690 can be moved, e.g. by a user, optionally assistedby one or more motors, controllers, drives, hydraulic and/or electricsystem, and/or robotic components and/or arms. Optionally, the imagingsystem, in this non-limiting example a CT, cone beam CT, spiral CT, MRIsystem, SPECT system, PET system 1780 or a combination thereof, and/orone or more of its components, and/or optionally the patient table 1690can be tracked, e.g. using any of the techniques described throughoutthe specification. In addition, the patient 1650 and/or the target organor target anatomic structure, such as a portion of a spine 1710, canalso optionally be tracked, e.g. using any of the techniques describedthroughout the specification. The system, including one or more of itscomputing systems and/or computer processors, can be configured so thatthe 3D stereoscopic view or augmented view of the 3D representation 1700of the surface or volume of the envelope, perimeter, limits, margin, orcombination thereof of the intended or desired imaging data acquisitiondisplayed by the HMD or other augmented reality device at a definedposition/orientation relative to one or more components of the imagingsystem, optionally superimposed onto the patient, prior to the actualimaging data acquisition can move in relation with the tracked imagingsystem 1780, tracked one or more components of the imaging system,(optionally tracked) patient table 1690, and/or the anatomic targetstructure, such as a spine 1710 (optionally also tracked, e.g. with oneor more attached markers or fiducials or fiducial arrays, not shown).When the imaging system 1780 or one or more of its components are moved,or when the patient table 1690 is moved, or when one or more parametersdetermining and/or influencing the location and/or orientation of theimage acquisition are changed, or any combination thereof, the systemcan be configured to adjusting the position, orientation, position andorientation of the 3D stereoscopic view or augmented view of the 3Drepresentation 1700 in response to the movement of the tracked imagingsystem 1750, the movement of the one or more of its components, themovement of the optionally tracked patient table 1690, and/or thechanges in the one or more parameters determining and/or influencing thelocation and/or orientation of the image acquisition.

FIGS. 8C-8E show how the 3D representation 1700 of the surface or volumeof the envelope, perimeter, limits, margin, or combination thereof ofthe intended or desired imaging data acquisition displayed by the HMD orother augmented reality device in a defined position, orientation orcombination thereof relative to the one or more components of theimaging system prior to the actual imaging data acquisition, optionallysuperimposed onto the patient, moves in relationship to a targetanatomic structure, such as a spine 1710 with the movement of thetracked imaging system 1780, the movement of the one or more of itscomponents, the movement of the optionally tracked patient table 1690,and/or the changes in the one or more parameters determining and/orinfluencing the location and/or orientation of the image acquisition.

In FIG. 8C, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed, by the HMD orother augmented reality display device, onto the patient 1650 prior tothe actual imaging data acquisition is too far anterior relative to atarget anatomic structure, such as a spine 1710. This can occur, forexample, when the imaging system 1780 or one or more of its componentsare too low in position relative to the patient 1650 or the targetanatomic structure, such as a spine 1710, or when the patient table 1690is too high relative to the imaging system 1780. An exploded view 1720,shows what the images will look like after the image data acquisition.Only a small anterior portion of the spine 1730 will be included in theactual image data volume acquisition and the resultant images.

In FIG. 8D, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed, by the HMD orother augmented reality display device, onto the patient 1650 prior tothe actual imaging data acquisition is too far posterior relative to atarget anatomic structure, such as a spine 1710. The 3D representation1700 is partially floating above the patient 1650. This can occur, forexample, when the imaging system 1780 or one or more of its componentsare too high or mal-aligned in position relative to the patient 1650 orthe target anatomic structure, such as a spine 1710, or when the patienttable 1690 is too low relative to the imaging system 1780, e.g. thecenter of the imaging system, the center of a rotation of a C-arm, thecenter of a spin of a cone beam CT, and/or the center of a spiral, e.g.with a CT scan. An exploded view 1720, shows what the images will looklike after the actual image data acquisition. Only a small posteriorportion of the spine 1730 will be included in the image data volumeacquisition and the resultant images.

In FIG. 8E, the 3D representation 1700 of the surface or volume of theenvelope, perimeter, limits, margin, or combination thereof of theintended or desired imaging data acquisition superimposed, by the HMD orother augmented reality display device, onto the patient 1650 prior tothe actual imaging data acquisition is correctly positioned relative toa target anatomic structure, such as a spine 1710. This can occur, forexample, when the imaging system 1780 or one or more of its componentsare correctly positioned, e.g. centered relative to the patient 1650 orthe target anatomic structure, such as a spine 1710, and/or when thepatient table 1690 is correctly positioned relative to the imagingsystem 1780, and/or when the patient 1650 is correctly positioned on thepatient table 1690, e.g. in the center of the patient table, not off tothe side. An exploded view 1720, shows what the images will look likeafter actual the image data acquisition. The entire target portion ofthe spine 1730 will be included in the image data volume acquisition andthe resultant images.

Thus, by moving one or more of the tracked imaging system 1780, trackedone or more components of the imaging system, (optionally tracked)patient table 1690, and/or the patient 1650, and/or the (optionallytracked) anatomic target structure, such as a spine 1710 prior to theactual image data acquisition and by optimizing the position,orientation, or position and orientation and/or coordinates of the 3Drepresentation 1700 of a surface or volume of the envelope, margin,perimeter and/or limits of the 3D imaging data volume acquisitionintended or desired for a patient 1650, and/or the target anatomictarget structure, such as a spine 1710, displayed by the HMD or otheraugmented reality device prior to the actual image data acquisition, theactual image data acquisition can be optimized with regard to thecoverage and/or inclusion of the target anatomic target structure, suchas a spine 1710. This approach can obviate the need for repeat scoutimages, e.g. MRI scout images, CT scout images, and/or x-ray images(e.g. AP, lateral) prior to the actual 3D volume acquisition.

Calibration Phantoms for Determining the Location of a 3D Representationof a Surface, Volume, Envelope or Perimeter of an Intended ImageAcquisition in Relationship to One or More Components of an ImagingSystem for Various Imaging System Settings

In some embodiments, an imaging calibration phantom can be used fordetermining the location of a 3D representation of a surface or volumecontaining information about an envelope, limit, perimeter, and/orboundary of an intended imaging acquisition in relationship to one ormore components of an imaging system. The imaging calibration phantomcan, for example, comprise one or more image visible markers, e.g.spheres, components, or compartments, containing various type ofmaterials depending on the imaging modality used. For example, forimaging systems using x-rays or x-ray beams, the materials can comprisea metal, e.g. aluminum, titanium, steel, lead or any other metal. Forimaging systems utilizing radionuclides, the phantom can comprise imagevisible markers, e.g. spheres, compartments or containers holding,containing one or more radioisotopes or radionuclides. The phantom cancomprise spheres, compartments or containers holding, containingcontrast media suitable for a particular imaging modality or imagingsystem. For example, for MRI, the phantom can comprise spheres,compartments or containers containing gadolinium-DTPA-doped water;alternatively, oil based fluids can also be used.

The phantom can comprise defined geometric patterns or arrangements ofimage visible markers within a single plane or within multiple planes.For example, for imaging modality using x-rays, one or more metalspheres or beads, i.e. image visible markers, can be arranged in a 3Dlayout, with multiple different spheres or beads located in differentlocations, each with different x, y, and z-coordinates, for example inmultiple layers.

The phantom and the image visible markers, e.g. spheres, beads etc.contained therein can be smaller, similar in size, or larger than thesurface, volume, envelope, limit, perimeter of an intended imagingacquisition. For example, the phantom can be larger than the maximumenvelope, limit, perimeter and/or boundary of the imaging acquisition.The image visible markers can be closely spaced in 3 dimensions, e.g. at20 mm, 15 mm, 10 mm, 5 mm, 3 mm, 2 mm, 1 mm intervals or any otherinterval. The phantom can be tracked using any of the tracking orregistration techniques known in the art or described in thespecification. For example, the phantom can comprise one or morefiducial markers, optical markers, fiducial arrays etc.

In some embodiments, the phantom can be placed on a table associatedwith the imaging system, e.g. a patient table, located in the generalarea and/or opening of the imaging system or imaging system components,e.g. a C-arm or O-arm, CT scanner, cone beam CT or other imaging system.The system can be configured to track a head mounted display, anaugmented or mixed reality display device, a patient table used with theimaging system, an imaging system, one or more components of the imagingsystem components, or a combination thereof and the phantom. Thegeometry of the image visible markers within the phantom or integratedor attached to the phantom can be known based on the manufacturingspecification and/or an optional post-manufacturing imaging test.

A computer processor can be configured to detect the image visiblemarkers of the phantom automatically, for example using automated imagesegmentation software. The phantom with the image visible markers can beplaced on the patient table of the imaging system. The location,position, and/or orientation of the phantom can be determined using thetracking system in relationship to the tracked imaging system or trackedone or more components of the imaging system. An image acquisition canbe performed; the computer processor can detect the image visiblemarkers included in the image acquisition. Thus, the computer processorcan determine which image visible markers were included in the imageacquisition and which of the image visible markers of the phantom werenot included; the boundary of the image visible markers included in theimage(s)/image volume and the image visible markers not included in theimage(s)/image volume can be used by the computer processor to determinethe boundary, limit, perimeter, envelope, surface or volume of the imageacquisition, e.g. in relationship to the tracked phantom and, with that,in relationship to the tracked one or more components of the imagingsystem. The one or more components of the imaging system can be trackedusing extrinsic or intrinsic tracking information and/or techniques, asdescribed throughout the specification. Someone skilled in the art canrecognize that the definition of the boundary, limit, perimeter,envelope of the image acquisition, used for the generation of the 3Drepresentation of the surface or volume of the image acquisition, can beimproved the more closely the image visible markers are spaced togetherwithin the phantom. In some embodiments, the phantom can be larger thanthe largest image acquisition volume provided by the imaging system.

With the location, position, and/or orientation of the tracked phantomin relationship to the tracked imaging system or tracked one or morecomponents of the imaging system known based on the trackinginformation, and the geometry of the image visible markers of thephantom known, and/or the image visible markers detected in the imageacquisition data/volume known, a computer processor can generate anestimate of a surface or volume of the boundary, limit, perimeter,and/or envelope of the image acquisition.

In some embodiments, the image acquisition can be repeated for differentgeometries of the imaging system (e.g. different tube detectordistances), different patient table heights, different positions,orientations and/or geometries of one or more components of the imagingsystem (e.g. a collimator), different geometries of the imageacquisition, and/or different image acquisition parameters. Using theforegoing techniques for determining the surface or volume of theboundary, limit, perimeter, and/or envelope of the image acquisition,the data and corresponding 3D representations of the surface or volumeof the boundary, limit, perimeter, and/or envelope of the imageacquisitions can be stored for a given imaging system and differentgeometries, positions, orientations, and/or image acquisitionparameters.

When a patient is subsequently placed on the imaging system table, a 3Drepresentation of the surface or volume of the boundary, limit,perimeter, and/or envelope of the image acquisition can be generatedand/or displayed corresponding to or reflecting a set of geometries,positions, orientations, and/or image acquisition parameters selectedfor that patient and the intended scan. The system can then display, byan HMD or other augmented reality display device, the 3D representationin relationship to the one or more imaging components; the imagingsystem can be moved to the target anatomic area of the patient, whilemaintaining the display of the 3D representation in relationship to theone or more imaging components. The imaging system can be moved to anoptimized position for a subsequent image acquisition relative to thetarget anatomic area, as seen in the stereoscopic view or augmented viewby the position of the 3D representation in relationship to the targetanatomic area.

Someone skilled in the art can recognize that data or informationrelated to the surface or volume of the boundary, limit, perimeter,and/or envelope of am image acquisition can also be retrieved from animaging system profile or database provided by an imaging systemmanufacturer; the surface or volume of the boundary, limit, perimeter,and/or envelope of am image acquisition and the 3D representations andcan be automatically updated and/or adjusted based on intrinsic imagingsystem information, e.g. different geometric settings of the imagingsystem (e.g. different tube detector distances), different positions,orientations and/or geometric settings of one or more components of theimaging system (e.g. a collimator), different geometric settings of theimage acquisition, and/or different image acquisition parameters.

EXAMPLES Example 1— Use of Multiple Head Mounted Displays

In some embodiments, multiple head mounted displays can be used. Headmounted displays (HMD) can be video see-through head mounted displays oroptical see-through head mounted displays. Referring to FIG. 9, a system10 for using multiple HMDs 11, 12, 13, 14 or other augmented realitydisplay systems for multiple viewer's, e.g. a primary surgeon, secondsurgeon, surgical assistant(s) and/or nurses(s) is shown; videosee-through head mounted displays could also be used in any of theembodiments. The multiple HMDs or other augmented reality displaysystems can be registered in a common coordinate system 15 usinganatomic structures, anatomic landmarks, calibration phantoms, referencephantoms, optical markers, navigation markers, and/or spatial anchors,for example like the spatial anchors used by the Microsoft Hololens.Pre-operative data 16 of the patient can also be registered in thecommon coordinate system 15. Live data 18 of the patient, for examplefrom the surgical site, e.g. a spine, optionally with minimally invasiveaccess, a hip arthrotomy site, a knee arthrotomy site, a bone cut, analtered surface can be measured, for example using one or more IMU's,optical markers, navigation markers, image or video capture systemsand/or spatial anchors. The live data 18 of the patient can beregistered in the common coordinate system 15. Intra-operative imagingstudies 20 can be registered in the common coordinate system 15. ORreferences, e.g. an OR table or room fixtures can be registered in thecommon coordinate system 15 using, for example, optical markers IMU's,navigation markers or spatial mapping 22. The pre-operative data 16 orlive data 18 including intra-operative measurements or combinationsthereof can be used to develop, generate or modify a virtual surgicalplan 24. The virtual surgical plan 24 can be registered in the commoncoordinate system 15. The HMDs 11, 12, 13, 14 or other augmented realitydisplay systems can project digital holograms of the virtual data orvirtual data into the view of the left eye using the view position andorientation of the left eye 26 and can project digital holograms of thevirtual data or virtual data into the view of the right eye using theview position and orientation of the right eye 28 of each user,resulting in a shared digital holographic experience 30. Using a virtualor other interface, the surgeon wearing HMD 1 11 can execute commands32, e.g. to display the next predetermined bone cut, e.g. from a virtualsurgical plan or an imaging study or intra-operative measurements, whichcan trigger the HMDs 11, 12, 13, 14 to project digital holograms of thenext surgical step 34 superimposed onto and aligned with the surgicalsite in a predetermined position and/or orientation.

Virtual data of the patient can be projected superimposed onto live dataof the patient for each individual viewer by each individual HMD fortheir respective view angle or perspective by registering live data ofthe patient, e.g. the surgical field, and virtual data of the patient aswell as each HMD in a common, shared coordinate system. Thus, virtualdata of the patient including aspects of a virtual surgical plan canremain superimposed and/or aligned with live data of the patientirrespective of the view angle or perspective of the viewer andalignment and/or superimposition can be maintained as the viewer moveshis or her head or body.

Example 2— Optional Imaging Data or Other Data Segmentation

When images of the patient are superimposed onto live data seen throughthe optical head mounted display, in many embodiments image segmentationcan be desirable. Any known algorithm in the art can be used for thispurpose, for example thresholding, seed point techniques, live wire,deformable models, statistical models, active shape models, level setmethods, marching cubes algorithms, artificial neural networks, deeplearning techniques, or combinations thereof and the like. Many of thesealgorithms are available is part of open-source or commercial libraries,for instance the Insight Segmentation and Registration Toolkit (ITK),the Open Source Computer Vision Library OpenCV, G'MIC (GREYC's Magic forImage Computing), Caffe, or MATLAB (MathWorks, Natick, Mass.). Arepresentative workflow for segmentation and subsequent is provided inFIG. 10. An optional pre-operative imaging study 40 can be obtained. Anoptional intra-operative imaging study 41 can be obtained. Thepre-operative 40 or intra-operative 41 imaging study can be segmented42, extracting, for example, surfaces, volumes or key features. Anoptional 3D reconstruction or 3D rendering 43 can be generated. Thepre-operative 40 or intra-operative 41 imaging study and any 3Dreconstruction or 3D rendering 43 can be registered in a commoncoordinate system 44. The pre-operative 40 or intra-operative 41 imagingstudy and any 3D reconstruction or 3D rendering 43 can be used forgenerating a virtual surgical plan 45. The virtual surgical plan 45 canbe registered in the common coordinate system 44. The surgical site 46can be registered in the common coordinate system 44. Intra-operativemeasurements 47 can be obtained and can be used for generating a virtualsurgical plan 45. A head mounted display 48 can project or displaystereoscopic images of virtual data or virtual data 49 superimposed ontoand aligned with the surgical site. The HMD 48 can be configured to usea built-in camera or image capture or video capture system 50 tooptionally detect and/or measure the position and/or orientation and/oralignment of one or more optical markers 51, which can be used for thecoordinate measurements 52, which can be part of the intra-operativemeasurements 47.

Example 3—Registration and Optional Re-Registration of StereoscopicDisplays

FIG. 11 illustrates an example of registering a stereoscopic display(e.g. digital hologram) or virtual data for an initial surgical step,performing the surgical step and re-registering one or more hologramsfor subsequent surgical steps. An optical head mounted display canproject or display a digital hologram of virtual data or virtual data ofthe patient 55. The stereoscopic display/digital hologram can optionallybe fixed to the HMD so that it will move with the movement of the HMD56. The operator can move the HMD until digital hologram of the virtualdata or virtual data of the patient is superimposed and aligned with thelive data of the patient, e.g. the surgical site 57. The digitalhologram of the virtual data or virtual data can then be registeredusing the same or similar coordinates as those of the live data withwhich the digital hologram is superimposed 58. The surgeon can thenperform one or more predetermined surgical steps, e.g. bone cuts 59. Adigital hologram of the virtual data or virtual data can optionally beregistered or re-registered after the surgical alteration with the livedata 60. The digital hologram of the virtual data or virtual data afterthe surgical alteration can optionally be displayed by the HMD 61. Thedigital hologram of the virtual data or virtual data after the surgicalalteration can optionally be fixed relative to the HMD so that it willmove with the movement of the HMD 62. The operator can move the HMDuntil digital hologram of the virtual data or virtual data of thepatient after the surgical alteration is superimposed and aligned withthe live data of the patient after the surgical alteration 63. Thedigital hologram of the virtual data or virtual data can then beregistered using the same or similar coordinates as those of the livedata after the surgical alteration with which the digital hologram issuperimposed 64. The surgeon can then perform one or more predeterminedsubsequent surgical steps, e.g. bone cuts, milling or drilling 65. Thepreceding steps can optionally be repeated until the surgical proceduresare completed 66. A virtual surgical plan 67 can be utilized.Optionally, the native anatomy of the patient including after a firstsurgical alteration can be displayed by the HMD 68. The HMD canoptionally display stereoscopic displays/digital holograms of subsequentsurgical steps 69.

Example 4—Display of Virtual Objects, Exemplary Applications for VirtualInterfaces, e.g. for Operating Robots and/or Imaging Systems

In some embodiments, the HMD can display a virtual object, e.g. anarbitrary virtual plane over the surgical field. The virtualobject/arbitrary virtual plane can be moveable using a virtual or otherinterface. For example, the virtual object/arbitrary virtual plane caninclude a “touch area”, wherein gesture recognition software, forexample the one provided by Microsoft with the Microsoft Hololensincluding, for example, the integrated virtual “drag function” forholograms can be used to move the arbitrary virtual plane. For example,one or more cameras integrated or attached to the HMD can capture themovement of the surgeon's finger(s) in relationship to the touch area;using gesture tracking software, the virtual object/virtual plane canthen be moved by advancing the finger towards the touch area in adesired direction. The movement of the virtual object/virtual plane viathe user interaction, e.g. with gesture recognition, gaze tracking,pointer tracking etc., can be used to generate a command by a computerprocessor. The command can trigger a corresponding movement of one ormore components of a surgical robot and/or an imaging system.

The HMD can display the virtual object/arbitrary virtual plane in anylocation initially, e.g. projected onto or outside the surgical field,e.g. a hip joint, knee joint, shoulder joint, ankle joint, or a spine.The HMD can optionally display the virtual object/arbitrary virtualplane at a defined angle, e.g. orthogonal or parallel, relative to afixed structure in the operating room, which can, for example, berecognized using one or more cameras, image capture or video capturesystems and/or a 3D scanner integrated into the HMD and spatialrecognition software such as the one provided by Microsoft with theMicrosoft Hololens or which can be recognized using one or more attachedoptical markers or navigation markers including infrared or RF markers.For example, one or more optical markers can be attached to an extensionof the operating table. The HMD can detect these one or more opticalmarkers and determine their coordinates and, with that, the horizontalplane of the operating room table. The virtual object/arbitrary virtualplane can then be displayed perpendicular or at another angle relativeto the operating room table. For example, in a hip replacement, the HMDcan display a virtual arbitrary plane over the surgical site. Thevirtual arbitrary plane can be perpendicular to the operating table orat another predefined or predetermined angle relative to the OR table.Using a virtual interface, e.g. a touch area on the virtual surgicalplane and gesture tracking, the HMD can detect how the surgeon is movingthe virtual arbitrary plane. Optionally, the virtual arbitrary plane canmaintain its perpendicular (or of desired other angle) orientationrelative to the OR table while the surgeon is moving and/or re-orientingthe plane; a perpendicular orientation can be desirable when the surgeonintends to make a perpendicular femoral neck cut. A different angle canbe desirable, when the surgeon intends to make the femoral neck cut withanother orientation. The position and/or orientation of the virtualobject can be transmitted from a second computing system incommunication with the HMD to a first computing system, e.g. incommunication with a surgical robot and/or an imaging system. Theposition and/or orientation of the virtual object can be used to setcoordinates for an image acquisition, e.g. in an area or volume definedin relationship to the virtual object. The position and/or orientationof the virtual object can be used to set coordinates for a boneresection by a surgical robot, e.g. with an end effector, wherein theend effector can comprise a pin, drill, mill, saw, saw blade, reamer,impactor or a combination thereof.

Using the touch area or other virtual interface, the surgeon can movethe virtual object, e.g. arbitrary virtual plane, into a desiredposition, orientation and/or alignment. The moving of the arbitraryvirtual plane can include translation and rotation or combinationsthereof in any desired direction using any desired angle or vector,which can be transmitted wirelessly from the second to the firstcomputing system and which can be used to generate one or more commands,for example for moving, aligning, positioning and/or orienting an endeffector, a surgical robot, an imaging system or a combination thereof.The surgeon can move the virtual object, e.g. arbitrary virtual plane tointersect with select anatomic landmarks or to intersect with selectanatomic or biomechanical axes. The surgeon can move the virtual object,e.g. arbitrary virtual plane to be tangent with select anatomiclandmarks or select anatomic or biomechanical axes.

For example, in a hip replacement, the surgeon can move the arbitraryvirtual plane to be tangent with the most superior aspect of the greatertrochanter and the most superior aspect of the lesser trochanter. FIG.12A shows an illustrative example of a virtual plane 70 that a primarysurgeon has moved and aligned to be tangent with the most superioraspect of the greater trochanter 71 and the most superior aspect of thelesser trochanter 72. FIG. 12B shows an illustrative example of the samevirtual plane 70 that the primary surgeon has moved and aligned to betangent with the most superior aspect of the greater trochanter 71 andthe most superior aspect of the lesser trochanter 72, now with the viewfrom the optical head mounted display of a second surgeon or surgicalassistant, e.g. on the other side of the OR table.

Optionally, for example with a pointer with an attached optical markeror an attached navigation marker, or with his finger detected using animage or video capture system integrated into the HMD and gesturerecognition software such as the one provided by Microsoft with theHololens, or with his finger with an attached optical marker ornavigation marker, the surgeon can point at and identify the sulcuspoint, e.g. the lowest point between the greater trochanter and thefemoral neck, which can be an additional reference. The line connectingthe most superior aspect of the greater trochanter and the most superioraspect of the lesser trochanter can then be determined on apre-operative or intra-operative AP radiograph of the hip; optionally,the sulcus point can also be detected on the AP radiograph. The APradiograph can include a template used by the surgeon for selecting andsizing, for example, the femoral and acetabular component, as well asthe liner and/or femoral heads. The radiographic template can include anindication for the femoral neck cut. The angle between the lineconnecting the most superior aspect of the greater trochanter and themost superior aspect of the lesser trochanter and the indication for thefemoral neck cut can be determined. FIG. 12C is an illustrative examplethat shows that a second virtual plane 73, the virtual femoral neck cutplane 73, can then be projected or displayed by the HMD, alsoperpendicular to the OR table like the arbitrary virtual plane 70, thelatter tangent with the most superior aspect of the greater trochanter71 and the most superior aspect of the lesser trochanter 72, and thefemoral neck cut plane 73 at the same angle and/or distance to thearbitrary virtual plane as the angle and distance between the lineconnecting the most superior aspect of the greater trochanter and themost superior aspect of the lesser trochanter and the indication for thefemoral neck cut on the radiograph. In this manner, the femoral neck cutplane can be defined using a second virtual plane prescribed orpredetermined based on the intra-operatively placed arbitrary virtualplane, moved by the operator to be tangent with the most superior aspectof the greater trochanter and the most superior aspect of the lessertrochanter. The second virtual plane, in this example, can be thefemoral neck cut plane which can be transmitted wirelessly from a secondcomputing system in communication with an HMD to a first computingsystem in communication with a surgical robot; a computer processor canbe configured to generate one or more commands to move and/or adjust theend effector of the surgical robot to align with the virtual femoralneck cut plane and to subsequently execute the femoral cut, superimposedand aligned with the virtual femoral neck cut plane.

The virtual femoral neck cut plane prescribed and projected or displayedin this manner can also be a virtual guide, e.g. a virtual cut blockthat projects, for example, a virtual slot for guiding a physical saw.The virtual guide or virtual cut block can have one or more dimensionsidentical to a physical guide or cut block, so that the physical guideor cut block can be aligned with the virtual guide or cut block. Thevirtual guide or cut block can be an outline, 2D or 3D, partial orcomplete, of the physical guide or cut block, with one or more identicaldimensions, so that the surgeon can align the physical guide or cutblock with the virtual guide or cut block. The virtual guide or cutblock can include placement indicia for the physical guide or cut block.

Example 5— Use of One or More/Multiple HMDs or Other Augmented RealityDisplay Systems for Modifying Virtual Surgical Plans and/or forOperating a Surgical Robot or an Imaging System

FIG. 13 shows an illustrative example how multiple HMDs or otheraugmented reality display systems can be used during a surgery, forexample by a first surgeon, a second surgeon, a surgical assistantand/or one or more nurses and how a surgical plan, e.g. for use with asurgical robot, can be modified and displayed during the procedure bymultiple HMDs or other augmented reality display systems whilepreserving the correct perspective view of virtual data andcorresponding live data for each individual operator. A system 10 forusing multiple HMDs 11, 12, 13, 14 or other augmented reality displaysystems for multiple viewer's, e.g. a primary surgeon, second surgeon,surgical assistant(s) and/or nurses(s) is shown. The multiple HMDs orother augmented reality display systems can be registered in a commoncoordinate system 15 using anatomic structures, anatomic landmarks,calibration phantoms, reference phantoms, optical markers, navigationmarkers, scanners, cameras, 3D scanners, and/or spatial anchors, forexample like the spatial anchors used by the Microsoft Hololens.Pre-operative data 16 of the patient can also be registered in thecommon coordinate system 15. Live data 18 of the patient, for examplefrom the surgical site, e.g. a spine, optionally with minimally invasiveaccess, a hip arthrotomy site, a knee arthrotomy site, a bone cut, analtered surface can be measured, for example using one or more IMU's,optical markers, navigation markers, image or video capture systemsand/or 3D scanner and/or spatial anchors. The live data 18 of thepatient can be registered in the common coordinate system 15.Intra-operative imaging studies 20 can be registered in the commoncoordinate system 15. OR references, e.g. an OR table or room fixturescan be registered in the common coordinate system 15 using, for example,optical markers IMU's, navigation markers or spatial mapping 22. Thepre-operative data 16 or live data 18 including intra-operativemeasurements or combinations thereof can be used to develop, generate ormodify a virtual surgical plan 24. The virtual surgical plan 24 can beregistered in the common coordinate system 15. The HMDs 11, 12, 13, 14or other augmented reality display systems can maintain alignment andsuperimposition of virtual data of the patient and live data of thepatient for each HMD 11, 12, 13, 14 for each viewer's perspective viewand position and head position and orientation 27. Using a virtual orother interface, the surgeon wearing HMD 1 11 can interact with one ormore virtual objects displayed by the HMDs or other augmented realitydisplay systems to generate/execute commands 32, e.g. to display thenext predetermined bone cut, e.g. from a virtual surgical plan or animaging study or intra-operative measurements, which can trigger theHMDs 11, 12, 13, 14 or other augmented reality display systems toproject virtual data of the next surgical step 34 superimposed onto andaligned with the surgical site in a predetermined position and/ororientation. One or more computing systems/computer processors incommunication with the HMDs or other augmented reality display systemscan generate the commands triggered by the user interaction with thevirtual object(s) displayed by the HMD. The one or more computingsystems/computer processors can optionally transmit the commandswirelessly to one or more different computing systems/computerprocessors, e.g. in communication with a robot and/or an imaging system.The one or more different computing systems/computer processors, e.g. incommunication with a robot and/or an imaging system can, for example,process the command(s) to initiate, start, stop, activate, de-activate,move, adjust the position/orientation of one or more components,controllers, drivers, motors, sensors, relays, processors or combinationthereof of a surgical robot and/or an imaging system. Any of the HMDs11, 12, 13, 14 or other augmented reality display systems can acquireone or more optical or other measurements or measurement inputs, e.g. ofanatomic landmarks, axes from cameras, anatomic axes, biomechanicalaxes, a mechanical axis of a leg 17, using for example an integrated orattached camera, image capture or video system or scanner, e.g. 3Dscanner. By using multiple HMDs 11, 12, 13, 14 or other augmentedreality display systems from different view angles with multiplecameras, image capture or video systems, scanners, 3D scanners theaccuracy of the measurements can optionally be improved. Optionally,parallax/stereoscopic measurements can be performed using the multipleHMDs 11, 12, 13, 14 or other augmented reality display systems fromdifferent view angles with multiple cameras, image capture or videosystems. The one or more optical measurements can be used to modify thevirtual surgical plan 19, optionally using the information from multipleHMDs 11, 12, 13, 14 or other augmented reality display systems. Someoneskilled in the art can recognize that multiple coordinate systems can beused instead of a common coordinate system. In this case, coordinatetransfers can be applied from one coordinate system to anothercoordinate system, for example for registering the HMD, live data of thepatient including the surgical site, virtual instruments and/or virtualimplants and physical instruments and physical implants.

Example 6—HMD Display of Virtual Surgical Guides, e.g. Virtual Axes, forExample for Operating Surgical Robots and/or Imaging Systems

In a spine, a joint, e.g. a hip joint, one or more HMDs or otheraugmented reality display systems, one or more virtual data sets orvirtual data can be registered in a common coordinate system. In ajoint, e.g. a hip joint, two opposing articular surfaces, e.g. withopposing cartilage surfaces and underlying subchondral bone, can beregistered separately and/or optionally jointly in a coordinate system,e.g. a common coordinate system. A first articular surface can belocated on the pelvic side, i.e. on the acetabulum, a second articularsurface can be located on the proximal femur. Registering the firstarticular surface and/or or associated bones and/or structures and thesecond articular surface and/or or associated bones and/or structuresseparately in a common coordinate system can have the benefit ofallowing movement, e.g. flexion and/or extension and/or rotation and/orabduction, and/or adduction, and/or elevation and/or other movements,e.g. translation, of the first articular surface and/or or associatedbones and/or structures, e.g. on the acetabular side, in relationship tothe second articular surface and/or or associated bones and/orstructures, e.g. on the proximal femoral side, while maintainingregistration of the first articular surface and/or associated bonesand/or structures, e.g. on the acetabular side, and/or the secondarticular surface and/or or associated bones and/or structures, e.g. onthe proximal femoral side, e.g. in a common coordinate system or asub-coordinate system, for example optionally along with one or moreHMDs or other augmented reality display systems and/or optionally fixedstructures in the operating room, e.g. the OR table, and/or otherstructures or anatomic landmarks of the patient, e.g. irrespectivemovement of the individual portions of the joint; the foregoing appliesto any joint in the human body, e.g. a shoulder, elbow, wrist, finger,knee, ankle, foot or toe joint or a temporomandibular joint. In thismanner, the hip joint or any other joint can be placed in differentpositions, e.g. flexion, extension, rotation, abduction, adduction, e.g.a degree of hip abduction, e.g. 20, 30, 40 or other degrees, e.g. duringplacement of a femoral component, and a degree of hip abduction, e.g.30, 40, or 50 or other degrees, during placement of the acetabularcomponent, or any other degrees for either component placement dependingon surgical technique and surgeon preference, while the registration ofthe acetabular and/or the registration of the proximal femoral side andthe display of any virtual data, e.g. a virtual surgical guide, avirtual cut plane, a virtual implant component on the acetabular sideand/or the proximal femoral side can be maintained and superimposed ontothe corresponding anatomic area, e.g. the area intended for implantcomponent placement, irrespective of the movement of individual portionsof the joint, thereby allowing the one or more HMDs or other augmentedreality display systems to maintain anatomically registered displays ofvirtual data superimposed onto the corresponding portions of thephysical joint anatomy, e.g. an articular surface, including a normal,damaged and/or diseased cartilage and/or subchondral bone and/orcortical bone, e.g. in a tangent, intersecting and/or offset manner,e.g. external and/or internal to the normal, damaged and/or diseasedcartilage and/or subchondral bone and/or cortical bone.

FIGS. 14A-14F are illustrative examples of displaying a virtual surgicalguide, e.g. a virtual acetabular reaming axis, using one or more HMDs orother augmented reality display systems and aligning a physicalacetabular reamer, e.g. attached to or part of a surgical robot, withthe virtual reaming axis for placing an acetabular cup with apredetermined cup angle, offset, medial or lateral position and/oranteversion and/or inclination. FIG. 14A shows a first surgeon's view,e.g. through an HMD, onto the patient's exposed acetabulum 280. Notealso the anterior superior iliac spine 281 and the symphysis pubis 282,which can optionally be used for registration purposes, for exampleusing attached optical markers or navigation markers. In FIG. 14B, thefirst surgeon can see a virtual acetabular reaming axis 283 through theHMD, which can be oriented in a predetermined manner to achieve apredetermined acetabular cup angle, offset, medial or lateral positionand/or anteversion and/or inclination, e.g. from a virtual surgical planfor the patient. In FIG. 14C, the first surgeon aligns the physicalacetabular reamer shaft 284 so that its central axis is aligned orsuperimposed with the virtual acetabular reaming axis thereby placingthe reamer head 285 in the acetabulum in a predetermined position andorientation for a predetermined acetabular cup angle, offset, medial orlateral position and/or anteversion and/or inclination.

FIG. 14D shows a second surgeon's view with his or her respective viewperspective of live data and virtual data through the HMD onto thepatient's exposed acetabulum 280. Note also the anterior superior iliacspine 281 and the symphysis pubis 282, which can optionally be used forregistration purposes, for example using attached optical markers ornavigation markers. In FIG. 14E, the second surgeon can see the virtualacetabular reaming axis 283 through the HMD, which can be oriented in apredetermined manner to achieve a predetermined acetabular cup angle,offset, medial or lateral position and/or anteversion and/orinclination, e.g. from a virtual surgical plan for the patient. Thevirtual acetabular reaming axis is projected with a view angle or viewperspective matching the view angle or view perspective of the live dataof the patient seen by the second surgeon. In FIG. 14F, the secondsurgeon can see how the physical acetabular reamer shaft 284 is alignedby the first surgeon so that its central axis is aligned or superimposedwith the virtual acetabular reaming axis thereby placing the reamer head285 in the acetabulum in a predetermined position and orientation for apredetermined acetabular cup angle, offset, medial or lateral positionand/or anteversion and/or inclination. Optionally, a virtual interface,e.g. with one or more virtual objects, can be displayed by the HMD(s),for interaction by the surgeon(s). The interaction can be used togenerate, by a computer processor, a command which can be configured,for example, to initiate, start, stop, activate, de-activate, move,adjust the position/orientation of one or more components, controllers,drivers, motors, sensors, relays, processors or combination thereof of asurgical robot and/or an imaging system.

FIGS. 15A-15D provide an illustrative, non-limiting example of the useof virtual surgical guides such as a distal femoral cut block displayedby an HMD and physical surgical guides such as physical distal femoralcut blocks. FIG. 15A shows live data of a patient with a distal femur300 exposed during knee replacement surgery, a medial condyle 301, alateral condyle 302 and a trochlea 303. In FIG. 15B, one or more HMDs orother augmented reality display systems can display a virtual distalfemoral cut block, e.g. in a stereoscopic manner for the left eye andthe right eye of the surgeon(s) creating a form of electronic hologramof the virtual surgical guide, i.e. the virtual distal cut block. Thevirtual distal femoral cut block 304 in this example is an outline ofthe physical distal femoral cut block with substantially similardimensions as those of the physical distal femoral cut block. Thevirtual distal femoral cut block 304 is aligned based at least in parton coordinates of a predetermined position for guiding the distalfemoral cut, for example for achieving a predetermined varus or valguscorrection and/or a predetermined femoral component flexion relative tothe distal femur and, for example, its anatomic or biomechanical axes.In FIG. 15C, the physical surgical guide 305, i.e. the physical distalfemoral cut block 305 (solid line) in this example (which can beoptionally attached to or part of a surgical robot), can be moved andaligned (e.g. by the surgical robot) to be substantially superimposedwith or aligned with the virtual surgical guide 304, i.e. the virtualdistal femoral cut block 304 (broken line) in this example. The hiddenareas of the knee joint 306, obscured or hidden by the physical distalfemoral cut block 305, can optionally also be displayed by the HMD. InFIG. 15D, the physical distal femoral cut block 305 (which can beoptionally attached to or part of a surgical robot) can be attached tothe distal femoral bone using two pins 307. These pins 307 can be usedfor subsequent surgical steps, for example for referencing a flexion gapor an extension gap or for ligament balancing. The HMD can stop displaythe virtual surgical guide, i.e. the virtual distal femoral cut block inthis example, but can optionally continue display the hidden anatomy,e.g. hidden areas of the knee joint 306. Optionally, a virtualinterface, e.g. with one or more virtual objects, can be displayed bythe HMD(s), for interaction by the surgeon(s). The interaction can beused to generate, by a computer processor, a command which can beconfigured, for example, to initiate, start, stop, activate,de-activate, move, adjust the position/orientation of one or morecomponents, controllers, drivers, motors, sensors, relays, processors orcombination thereof of a surgical robot and/or an imaging system. Thecommand can, optionally, be transmitted from a computing system/computerprocessor in communication with the one or more HMDs or other augmentedreality display systems to a computing system/computer processor incommunication with a robot and/or an imaging system.

FIGS. 16A-16C provide an illustrative, non-limiting example of the useof virtual surgical guides such as an AP femoral cut block displayed byan HMD and physical surgical guides such as physical AP cut blocks forknee replacement. FIG. 16A shows live data of a patient with a distalfemur 300 exposed during knee replacement surgery after a distal femoralcut creating a planar distal surface 310, a medial condyle 301, alateral condyle 302 and a trochlea 303. In FIG. 16B, one or more HMDs orother augmented reality display systems can display a virtual femoral APcut block 312, e.g. in a stereoscopic manner for the left eye and theright eye of the surgeon(s) creating a form of electronic or digitalhologram of the virtual surgical guide, i.e. the virtual femoral AP cutblock 312. The virtual femoral AP cut block 312 in this example is anoutline of the physical femoral AP cut block with similar dimensions,edges, or planes as those of the physical femoral AP cut block. Thevirtual femoral AP cut block 312 is aligned based at least in part oncoordinates of a predetermined position for guiding the different bonecuts, e.g. an anterior cut, posterior cut and/or chamfer cuts dependingon the configuration of the physical femoral AP cut block, for examplefor achieving a predetermined femoral component rotation. In FIG. 16C,the physical surgical guide 314, i.e. the physical femoral AP cut block314 (solid line) in this example (e.g. attached to or part of a surgicalrobot), can be moved and aligned to be substantially superimposed withor aligned with the virtual surgical guide 312, i.e. the virtual femoralAP cut block 312 (broken line) in this example. The physical femoral APcut block (optionally attached to or part of a surgical robot) can beattached to the distal femoral bone using pins (not shown) and the cutscan be performed. Subsequent surgical steps can optionally be referencedbased on one or more of the cuts executed using the physical femoral APcut block.

The surgeon can align or substantially superimpose the physical femoralAP cut block with the digital hologram of the virtual femoral AP cutblock or its 2D or 3D outline or one or more placement indicatorsprojected by the HMD. Once adequate alignment or superimposition of thephysical AP cut block with the virtual AP cut block or its 2D or 3Doutline or one or more placement indicators displayed by the HMD hasbeen achieved, the surgeon can pin the physical AP cut block and performthe cuts. By utilizing preoperative 3D data information orintra-operative information, e.g. from optical marker and image or videocapture or scanner measurements (including stereoscopic imaging), forthe position, alignment and rotation of the physical femoral AP cutblock with the assistance of the HMD, the surgeon can perform theanterior and posterior femoral cuts in a highly accurate manner, therebyachieving accurate rotational alignment of the femoral component. Thesame approaches and display options, e.g. virtual cut blocks, 2D or 3Doutline or one or more placement indicators, can be applied to allsubsequent femoral preparation steps including chamfer cuts and chamfercut blocks.

Optionally, a virtual interface, e.g. with one or more virtual objects,can be displayed by the HMD(s), for interaction by the surgeon(s). Forexample, the virtual object can be a virtual surgical guide, e.g. thevirtual AP cut block. The interaction can comprise moving the virtualobject, e.g. the virtual surgical guide, for example using a trackedpointer, a gesture recognition, a finger tracking, an object tracking,etc. The interaction can be used to generate, by a computer processor, acommand which can be configured, for example, to initiate, start, stop,activate, de-activate, move, adjust the position/orientation of thevirtual object, e.g. virtual surgical guide, and, optionallycorrespondingly, one or more components, controllers, drivers, motors,sensors, relays, processors or combination thereof of a surgical robotand/or an imaging system. The command can, optionally, be transmittedfrom a computing system/computer processor in communication with the oneor more HMDs or other augmented reality display systems to a computingsystem/computer processor in communication with a robot and/or animaging system.

Of note, similar steps and HMD guided femoral procedures are alsopossible using the HMD with any of the other registration andcross-referencing techniques described in the present disclosure orknown in the art, for example intraoperative image guidance.

FIGS. 17A-17F provide an illustrative, non-limiting example of the useof virtual surgical guides such as a virtual proximal tibial cut guidedisplayed by an HMD and physical surgical guides such as physicalproximal tibial cut guide. FIG. 17A shows live data of a patient with aproximal tibia 330 exposed during knee replacement surgery, a medialtibial plateau 331, a lateral tibial plateau 332 and a medial tibialspine 333 and a lateral tibial spine 334. In FIG. 17B, one or more HMDsor other augmented reality display systems can display a virtualproximal tibial cut guide, e.g. in a stereoscopic manner for the lefteye and the right eye of the surgeon(s), creating a form of electronichologram of the virtual surgical guide, i.e. the virtual proximal tibialcut guide. The virtual proximal tibial cut guide 336 in this example canbe an outline of the physical proximal tibial cut guide withsubstantially similar dimensions as those of the physical proximaltibial cut guide. The virtual proximal tibial cut guide 336 is alignedbased at least in part on coordinates of a predetermined position forguiding the proximal tibial cut, for example for achieving apredetermined varus or valgus correction and/or a predetermined sloperelative to the proximal tibia and, for example, its anatomic orbiomechanical axes. In FIG. 17C, the physical surgical guide (e.g.integrated, attached to, or part of a surgical robot) 338, i.e. thephysical proximal tibial cut guide 338 (solid line) in this example, canbe moved and aligned to be substantially superimposed with or alignedwith the virtual surgical guide 336, i.e. the virtual proximal tibialcut guide 336 (broken line) in this example. Note two pin holes 339 inthe physical proximal tibial cut guide 338. In FIG. 17D, the physicalproximal tibial cut guide 338 can be attached to the proximal tibia boneusing two pins 340. These pins 307 can be used for subsequent surgicalsteps, for example for referencing a flexion gap or an extension gap orfor ligament balancing. In FIG. 17E, an alternative embodiment is shownto FIG. 17B. One or more HMDs or other augmented reality display systemscan display a virtual proximal tibial cut plane 342, e.g. in astereoscopic manner for the left eye and the right eye of thesurgeon(s), creating a form of electronic hologram of the virtual tibialcut plane. The virtual proximal tibial cut plane 342 in this example isparallel with and substantially aligned and superimposed with thepredetermined cut plane for the physical proximal tibial cut guide. Thevirtual proximal tibial cut plane 342 is aligned based at least in parton coordinates of a predetermined position for guiding the proximaltibial cut, for example for achieving a predetermined varus or valguscorrection and/or a predetermined slope relative to the proximal tibiaand, for example, its anatomic or biomechanical axes. A physical sawblade or a slot for aligning the physical saw blade (e.g. as an endeffector of a surgical robot with an integrated or attached saw) in aphysical proximal tibial cut guide or an open guide area foraccommodating the saw blade in a physical proximal tibial cut guide canthen be aligned and at least partially superimposed with the virtualproximal tibial cut plane 342. In FIG. 17F, an alternative embodiment isshown to FIG. 17B. One or more HMDs or other augmented reality displaysystems can display two or more virtual drills or pins 344 for placementin the proximal tibia, e.g. in a stereoscopic manner for the left eyeand the right eye of the surgeon(s), creating a form of electronichologram of the virtual tibial pins or drills. The virtual drills orpins 344 in this example can be an outline or a projected path of thephysical pins or drills that can be used to fixate a physical proximaltibial cut guide to the proximal tibia. The virtual drills or pins 344are aligned based at least in part on coordinates of a predeterminedposition for guiding the proximal tibial cut, for example for achievinga predetermined varus or valgus correction and/or a predetermined sloperelative to the proximal tibia and, for example, its anatomic orbiomechanical axes. The physical drills or pins (not shown) can then bealigned and superimposed with the virtual drills or pins 344 and placedin the proximal tibia. A physical proximal tibial cut guide can then beattached to the physical pins and the proximal tibial cut can beexecuted. Optionally, a virtual interface, e.g. with one or more virtualobjects, can be displayed by the HMD(s), for interaction by thesurgeon(s). For example, the virtual object can be a virtual surgicalguide, e.g. a virtual cut plane, a virtual pin, a virtual drill, avirtual axis, a virtual tool, a virtual instrument, a virtual implant.The interaction can comprise moving the virtual object, e.g. the virtualsurgical guide, for example using a tracked pointer, a gesturerecognition, a finger tracking, an object tracking, etc. The interactioncan be used to generate, by a computer processor, a command which can beconfigured, for example, to initiate, start, stop, activate,de-activate, move, adjust the position/orientation of the virtualobject, e.g. virtual surgical guide, e.g. the virtual cut plane, virtualpin, virtual drill, virtual axis, virtual tool, virtual instrument,virtual implant and, optionally correspondingly, one or more components,controllers, drivers, motors, sensors, relays, processors or combinationthereof of a surgical robot and/or an imaging system. The command can,optionally, be transmitted from a computing system/computer processor incommunication with the one or more HMDs or other augmented realitydisplay systems to a computing system/computer processor incommunication with a robot and/or an imaging system.

Example 7—Use of Markers, e.g. Optical Markers

In some embodiments, data can be obtained using an HMD, for example onemanufactured by Microsoft, the Microsoft Hololens or Hololens 2(Microsoft, Redmond, Wis.). The Hololens can use, for example, Windowsholographic APIs including Unity (Unity Technologies, San Francisco,Calif.) and Vuforia 6.2 (PTC, Inc., Needham, Mass.).

Registration of Optical Markers Using Microsoft Hololens and Vuforia 6.2

FIG. 18 shows a wooden board with 25 squares which was prepared and four4.0×4.0 cm optical markers 420 with four distinct QR codes were appliedin equidistant locations, 4.0 cm apart. As seen in FIG. 19, a softwareroutine was implemented to project four cubes 423 with dimensions of4.0×4.0×4.0 cm superimposed onto the squares and to maintainregistration over the squares irrespective of head movement. The resultsare shown in FIG. 19. The Microsoft Hololens was not able to maintainregistration of the four cubes over the designated optical markers; thecubes were at times displaced by as much as 3-4 cm and were also tilted.For example, registration of optical Markers using Hololens and OpenCV2.4 can be implemented. OpenCV 2.4, an open source computer visionframework (Intel Inc., Santa Clara, Calif.), can be implemented on theHololens system using OpenCVForUnity. As seen in FIG. 20, ArUco markers425 available with OpenCV with a size of 2.8×2.8 cm can be arranged at adistance of 3.0×3.0 cm. A cm scale 426 is shown at the bottom of FIG.20. Using this approach shown with the results shown in FIG. 21,acquisition of the 25 markers 425 using the internal Hololens camerarequired 1 second, corresponding to approximately 40 ms per marker.Markers were consistently recognized as indicated by the displayed greenmarker ID number 428, with only few occasional drop outs with no greenmarker ID number displayed 430 as seen in FIG. 21.

Markers can be mounted on a wooden board with a size of 2.8×2.8 cm andarranged at a distance of 3.0×3.0 cm and static measurements ofdisplacement of optically detected marker positions vs. actual markerpositions can be obtained at an angle of approximately 40 degreesbetween the Hololens and the board at a distance of approximately 32.5cm to the center of the board.

FIG. 22 shows an example comparing the actual marker dimensions (2.8×2.8cm) and position in black 432 with the optically detected marker usingthe Hololens camera seen as red outline 434. The marker is not square inthe image due to the angulation. The pixel size is approximately 0.5 mmin horizontal direction and 0.75 mm in vertical direction in this test.The data indicates sub-pixel accuracy which is why the followinganalysis of the data can be implemented: Pixels at the superior,inferior, left and right border are considered incorrectly detected ifmore than half have a grey value lower than the average grey value (i.e.the grey value between black and the grey background). For example, thehorizontal red line at the superior border in FIG. 22 would need to beexactly 1 pixel higher in order to be counted as correctly detected.Conversely, the inferior second and third horizontal red line from theleft are counted as accurately detected. The percentage of each edge(superior, inferior, left, right) that is correctly detected can thendetermined, e.g. 100% for the superior edge and 50% for the inferioredge in FIG. 22. The analysis over the 25 markers shows that the maximumdeviation between the optically detected marker position and the actualmarker is 0.75 mm, i.e. one pixel size in vertical direction, with anaverage deviation between the optically detected marker position and theactual marker of 0.349 pixel=0.26 mm.

Example 8—Applications of Virtual User Interfaces

A virtual interface for the path of a a pedicle screw, an interbodydevice, or other types of implant, e.g. for joint replacement, can use,for example, the Unity for HoloLens engine (Unity Technologies, SanFrancisco, Calif.). Unity's GestureRecognizer interface allows forrecognition of different hold, navigation and manipulation functions.Additionally, the Gaze functionality can be available for implementationof a cursor controlled by the user's view direction. Thus, in selectapplications, the user's gaze can control the cursor including cursormovement. Closure of the eye lid can, for example, also be used togenerate a command to execute a function. With the virtual interface,the planning can be performed, for example, on fluoroscopic imagesdisplayed by an HMD using gesture commands which are mapped to entrypoints and vectors. A vector corresponding to the intended path of thesurgical instrument(s), e.g. an awl or the pedicle screw, can be placedby the surgeon using gesture commands, e.g. a closed position of thethumb and index finger or an open position of a thumb and index fingeras shown in FIGS. 23A-23E. FIGS. 23A-23E show an illustrative examplefor placing an intended path of a pedicle screw. In FIGS. 23A-E, afluoroscopic image 440 of a lumbar spine 442 showing lumbar levels L1-L5is displayed by an HMD, registered with anatomic landmarks usingreference frame 444 with optical markers 446 attached to patient's back.Each marker has its own unique QR or other code, thereby optionallyidentifying the patient's left and right side, superior and inferior.Other markers, e.g. retroreflective marker balls or disks, can be used.In FIG. 23B, the HMD displays a preliminary path 448 in arbitrarylocation; thumb 450 and index finger 452 are also seen. The surgeon canmove the fingers towards the arbitrary path with fingers open. In FIG.23C, the surgeon closes thumb 450 and index fingers 452 over theintended path 454, triggering a command via gesture recognition to movethe intended path following the finger movement. In FIG. 23D, thesurgeon moves the intended path 455 into desired orientation over thepedicle by moving the thumb 450 and index finger 452. In FIG. 23E, thesurgeon opens the thumb 450 and index finger 452 triggering a commandvia gesture recognition to fixate the vector of the intended path 455 inthe coordinate system. An open position indicates in this non-limitingexample that the vector is anchored and the intended path is fixatedrelative to the global coordinate system and the anatomic landmarks. Aclosed position indicates in this non-limiting example that the vectorcan be moved with six degrees of freedom following the movement of thefingers. Any other finger symbols and movements can be used.

The user interaction, e.g. via gesture recognition or a virtual userinterface, for example displaying virtual objects such as virtualbuttons, sliders, keyboards, etc., can be used to generate, by acomputer processor, a command which can be configured, for example, toinitiate, start, stop, activate, de-activate, move, adjust theposition/orientation of the virtual object, e.g. virtual surgical guide,e.g. the virtual cut plane, virtual trajectory, virtual pin, virtualdrill, virtual axis, virtual tool, virtual instrument, virtual implantand, optionally correspondingly, one or more components, controllers,drivers, motors, sensors, relays, processors or combination thereof of asurgical robot and/or an imaging system. The command can, optionally, betransmitted from a computing system/computer processor in communicationwith the one or more HMDs or other augmented reality display systems toa computing system/computer processor in communication with a robotand/or an imaging system, which can then effect or cause the initiating,starting, stopping, activating, de-activating, moving, adjusting of theone or more components, controllers, drivers, motors, sensors, relays,processors or combination thereof of a surgical robot and/or an imagingsystem. For example, the command can cause the computing system/computerprocessor in communication with a robot to move a robotic arm and endeffector and/or a drill guide to align with a trajectory defined by thesurgeon using the virtual user interface and/or gesture recognition.

FIGS. 24A-24B provide illustrative, non-limiting examples of one or moreaugmented reality HMD displays including a virtual user interface 990,for example displaying one or more virtual button, virtual field,virtual cursor, virtual pointer, virtual slider, virtual trackball,virtual node, virtual numeric display, virtual touchpad, virtualkeyboard, or a combination thereof, for virtual placing, sizing,fitting, selecting and aligning of virtual pedicle screws and includingHMD displays for guidance of spinal instruments and implants. A virtualuser interface 990 can be configured for selecting different sizes ofvirtual pedicle screws, e.g. in mm of diameter. A computer processor canbe configured to allowing placing and moving of the virtual pediclescrews onto the virtually displayed spine 993 of the patient, e.g. usinga 3D model generated based on a pre-operative CT scan or anintra-operative O-arm or 3D C-arm scan. The computer processor can beconfigured for selecting different sizes of implants (e.g. in mm),using, for example, voice commands or gesture commands, e.g. a size 6.0mm. A virtual path 996 can be displayed for guiding the placement of theone or more physical pedicle screws. A computer processor can beconfigured to move, place, size, and align virtual pedicle screws 1000using, for example, gesture recognition or voice commands, and,optionally to display magnified views 1003, e.g. from a CT scan,demonstrating the pedicle 1006 including the medial wall of the pedicle1009. A target placement location 1012 for the virtual pedicle screw1000 can also be shown. The virtual screw can be adjusted to be placedin the center of the pedicle. The physical screw and/or awl or screwdriver can be tracked, e.g. using a navigation system or video system(for example with navigation markers or optical markers or directoptical tracking). When the screw path, awl path or screw driver pathextends beyond the medial wall of the pedicle, a computer processor cangenerate an alarm, e.g. via color coding or acoustic signals. Physicalinstruments, e.g. a physical awl 1015, can be aligned with andsuperimposed onto the virtual path 996 projected by an HMD.

The user interaction, e.g. via gesture recognition or a virtual userinterface, for example displaying virtual objects such as for exampleone or more virtual button, virtual field, virtual cursor, virtualpointer, virtual slider, virtual trackball, virtual node, virtualnumeric display, virtual touchpad, virtual keyboard, or a combinationthereof, can be used to generate, by a computer processor, a commandwhich can be configured, for example, to initiate, start, stop,activate, de-activate, move, adjust the position/orientation of thevirtual object, e.g. virtual surgical guide, e.g. the virtual cut plane,virtual trajectory, virtual pin, virtual drill, virtual axis, virtualtool, virtual instrument, virtual implant (e.g. a virtual screw orvirtual cage or other interbody device) and, optionally correspondingly,one or more components, controllers, drivers, motors, sensors, relays,processors or combination thereof of a surgical robot and/or an imagingsystem. The command can, optionally, be transmitted from a computingsystem/computer processor in communication with the one or more HMDs orother augmented reality display systems to a computing system/computerprocessor in communication with a robot and/or an imaging system, whichcan then effect or cause the initiating, starting, stopping, activating,de-activating, moving, adjusting of the one or more components,controllers, drivers, motors, sensors, relays, processors or combinationthereof of a surgical robot and/or an imaging system. For example, thecommand can cause the computing system/computer processor incommunication with a robot to move a robotic arm and end effector and/ora drill guide to align with a trajectory defined by the surgeon usingthe virtual user interface and/or gesture recognition.

FIGS. 25A-25B provide illustrative, non-limiting examples of one or moreaugmented reality HMD displays for virtual placing, sizing, fitting,selecting and aligning of implant components. A virtual femoralcomponent 960 can be displayed by one or more HMD displays. A virtualuser interface 963, for example displaying one or more virtual button,virtual field, virtual cursor, virtual pointer, virtual slider, virtualtrackball, virtual node, virtual numeric display, virtual touchpad,virtual keyboard, or a combination thereof, can be configured forselecting different sizes of virtual femoral components. A computerprocessor can be configured to allowing placing and moving of thevirtual femoral component onto the physical distal femur 966 of thepatient. The computer processor can be configured for selectingdifferent sizes of implants, using, for example, voice commands, e.g. asize 6, and for aligning the virtual femoral component 960 with thephysical distal femur of the live patient using gesture recognitionconfigured to recognize an index finger 969 and thumb 972, in theexample in FIG. 25B. The virtual implant can be registered and/ordisplayed in relationship to a coordinate system. One or more markers975, e.g. with QR codes or a retroreflective or other markers, can beregistered in the same coordinate system. The coordinates of the finalposition and/or orientation of the virtual implant, following adjustmentvia user interaction with the virtual user interface, can betransmitted, for example wirelessly, from a computing system/computerprocessor in communication with an HMD to a computing system/computerprocessor in communication with a surgical robot and/or imaging system.A command can be generated by one or more computer processors triggeredby user interaction with the virtual user interface and can betransmitted, for example wirelessly, from a computing system/computerprocessor in communication with an HMD to a computing system/computerprocessor in communication with a surgical robot and/or imaging system.The command can cause the computing system/computer processor incommunication with an imaging system to position and/or align theimaging system over the area, e.g. centered over the area, of thevirtual implant. The command can cause the computing system/computerprocessor in communication with a surgical robot to position and/oralign the surgical robot and/or its end effector over the area, e.g.centered over the area, of the virtual implant and to execute orfacilitate execution of bone resections for the placement of theimplant, e.g. a drilling, burring, cutting, milling, reaming and/orimpacting with one or more end effectors.

All publications, patents mentioned herein are hereby incorporated byreference in their entirety as if each individual publication or patentwas specifically and individually indicated to be incorporated byreference.

1. A system of preparing an imaging data acquisition associated with apatient comprising: at least one computer processor; an augmentedreality display device; and an imaging system, wherein the at least onecomputer processor is configured to obtain real-time trackinginformation of one or more components of the imaging system, wherein theat least one computer processor is configured to generate a 3Drepresentation of a surface, a volume or combination thereof, whereinthe 3D representation of the surface, volume or combination thereof isat least in part derived from information about a geometry of the one ormore components of the imaging system, information about a geometry ofthe image acquisition, information about one or more image acquisitionparameters, or a combination thereof, wherein the at least one computerprocessor is configured to generate an augmented view, the augmentedview comprising the 3D representation of the surface, volume orcombination thereof, wherein the at least one computer processor isconfigured to display, by the augmented reality display device, theaugmented view at a defined position and orientation relative to the oneor more components of the imaging system, and wherein the position andorientation of the augmented view is updated based on the real timetracking information of the one or more components of the imagingsystem.
 2. The system of claim 1, wherein the 3D representation of thesurface, volume or combination thereof does not contain imaging datafrom a patient.
 3. The system of claim 1, wherein the imaging system isconfigured to acquire 2D, 3D, or 2D and 3D imaging data of the patientwithin the 3D representation of the surface, volume or combinationthereof.
 4. The system of claim 1, wherein the surface, volume orcombination thereof comprises information about a limit, an edge, amargin, a boundary, a circumference, a perimeter, an envelope or acombination thereof of a 2D, 3D, or 2D and 3D imaging data acquisition.5. The system of claim 1, wherein the at least one computer processor isconfigured to adjust the augmented view responsive to movement of theone or more tracked components of the imaging system, wherein theadjustment is configured to maintain the augmented view at the definedposition and orientation relative to the one or more components of theimaging system.
 6. The system of claim 1, wherein the information aboutthe geometry of the imaging system, information about the geometry ofthe image acquisition, information about one or more image acquisitionparameter, or a combination thereof comprises information about one ormore imaging system components, a geometric relationship between one ormore imaging system components, a collimator, a grid, an imageintensifier, a detector resolution, an x-ray source, an x-ray tubesetting, a kVp setting, an mA setting, an mAs setting, a collimation, atube—detector distance, a tube—patient distance, patient—detectordistance, a patient—image intensifier distance, a table height relativeto a tube, a detector, a table position relative to a tube, a detector,or combination thereof, a patient position, a C-arm position,orientation, or combination thereof, a gantry position, orientation orcombination thereof, a grid height, a grid width, a grid ratio, a fieldof view, a center of a field of view, a periphery of a field of view, amatrix, a pixel size, a voxel size, an image size, an image volume, animaging plane, an image dimension in x, y, z and/or oblique direction,an image location, an image volume location, a scan coverage, a pitch,an in-plane resolution, a slice thickness an increment, a detectorconfiguration, a detector resolution, a detector density, a tubecurrent, a tube potential, a reconstruction algorithm, a scan range, ascan boundary, a scan limit, a rotational axis of the imaging system, arotational center of the imaging system, a reconstructed slicethickness, a segmentation algorithm, a window, a level, a brightness, acontrast, a display resolution, or a combination thereof.
 7. The systemof claim 1, wherein the imaging system comprises an x-ray system, afluoroscopy system, a C-arm, a 3D C-arm, a digital tomosynthesis imagingsystem, an angiography system, a bi-planar angiography system, a 3Dangiography system, a CT scanner, an MRI scanner, a PET scanner, a SPECTscanner, a nuclear scintigraphy system, a 2D ultrasound imaging system,a 3D ultrasound imaging system, or a combination thereof.
 8. The systemof claim 1, wherein the at least one computer processor is configured toobtain real-time tracking information of the augmented reality displaydevice, an anatomic structure of the patient, a patient table used withthe imaging system, the imaging system, the one or more components ofthe imaging system, or a combination thereof.
 9. The system of claim 8,further comprising a camera or scanner configured to acquire thereal-time tracking information of the augmented reality display device,the anatomic structure of the patient, the patient table used with theimaging system, the imaging system, the one or more components of theimaging system, or a combination thereof.
 10. The system of claim 1,wherein the system is configured to obtain real-time trackinginformation of the imaging system using intrinsic information from theimaging system, wherein the intrinsic information comprises pose data,sensor data, camera data, 3D scanner data, controller data, drive data,actuator data, end effector data, data from one or more potentiometers,data from one or more video systems, data from one or more LIDARsystems, data from one or more depth sensors, data from one or moreinertial measurement units, data from one or more accelerometers, datafrom one or more magnetometers, data from one or more gyroscopes, datafrom one or more force sensors, data from one or more pressure sensors,data from one or more position sensors, data from one or moreorientation sensors, data from one or more motion sensors, positionand/or orientation data from step motors, position and/or orientationdata from electric motors, position and/or orientation data fromhydraulic motors, position and/or orientation data from electric and/ormechanical actuators, position and/or orientation data from drives,position and/or orientation data from robotic controllers, positionand/or orientation data from one or more robotic computer processors, ora combination thereof.
 11. The system of claim 1, wherein the imagingsystem is configured to generate an x-ray beam, wherein the 3Drepresentation of the surface, volume or combination thereof comprisesinformation about a limit, an edge, a margin, a boundary, acircumference, a perimeter, an envelope or a combination thereof of thex-ray beam.
 12. The system of claim 1, the system further comprising auser interface.
 13. The system of claim 12, wherein the user interfacecomprises a virtual user interface, wherein the virtual interfacecomprises at least one virtual object.
 14. The system of claim 13,wherein the at least one virtual object comprises one or more virtualbutton, virtual field, virtual cursor, virtual pointer, virtual slider,virtual trackball, virtual node, virtual numeric display, virtualtouchpad, virtual keyboard, or a combination thereof.
 15. The system ofclaim 13, wherein the virtual user interface comprises a gesturerecognition, gaze recognition, gaze lock, eye tracking, hand tracking,pointer tracking, instrument tracking, tool tracking, or a combinationthereof.
 16. The system of claim 13, wherein the at least one computerprocessor is configured to generate a command based at least in part onat least one interaction of a user with the at least one virtual objectdisplayed in the virtual user interface.
 17. The system of claim 16,wherein the command is configured to move, tilt, or rotate one or morecomponents of the imaging system, one or more components of a patienttable or a combination thereof, or wherein the command is configured toactivate, operate, de-activate or a combination thereof a motor, anactuator, a drive, a controller, a hydraulic system, a switch, anelectronic circuit, a computer chip, an x-ray tube, an imageintensifier, a functional unit of an imaging system, or a combinationthereof, or wherein the command is configured to move or modify ageometry of the imaging system, a patient table, a geometricrelationship between one or more imaging system components, acollimator, a grid, an image intensifier, a detector resolution, asetting of the imaging system, a parameter of the imaging system, aparameter of the imaging data acquisition, a display parameter, an x-raysource setting, an x-ray tube setting, a kVp setting, an mA setting, anmAs setting, a collimation, a tube—detector distance, a tube—patientdistance, patient—detector distance, a patient—image intensifierdistance, a table height relative to a tube, a detector, a tableposition relative to a tube, a detector, a patient position, a C-armposition, orientation, or combination thereof, a gantry position,orientation or combination thereof, a grid height, a grid width, a gridratio, a field of view, a matrix, a pixel size, a voxel size, an imagesize, an image volume, an imaging plane, an image dimension in x, y, zand/or oblique direction, an image location, an image volume location, ascan coverage, a pitch, an in-plane resolution, a slice thickness, anincrement, a detector configuration, a detector resolution, a detectordensity, a tube current, a tube potential, a reconstruction algorithm, ascan range, a scan boundary, a scan limit, a reconstructed slicethickness, a segmentation algorithm, a window, a level, a brightness, acontrast, a display resolution, or a combination thereof, or wherein thecommand is configured to set and/or modify one or more image acquisitionparameters of the imaging system, or wherein the command is configuredto set, move, and/or modify a position, orientation, size, area, volume,or combination thereof of a 2D, 3D or 2D and 3D imaging dataacquisition.
 18. The system of claim 1, wherein the system is configuredto determine a desired location of the augmented view associated withthe imaging system to acquire 2D, 3D, or 2D and 3D imaging data at thedesired location.
 19. The system of claim 1, wherein the augmentedreality display device is a head mounted display, and wherein theaugmented view comprises a 3D stereoscopic view.
 20. The system of claim19, wherein the at least one computer processor is configured to projectthe 3D stereoscopic view at coordinates of intended 2D, 3D or 2D and 3Dimaging data acquisition of the patient.
 21. The system of claim 20,wherein location of the 2D, 3D, or 2D and 3D imaging data acquisitioncomprises one or more target anatomic structures of the patient.
 22. Amethod of preparing an image acquisition by an imaging system in apatient comprising: a. tracking one or more components of the imagingsystem in real time; b. obtaining, by at least one computer processor,information about a geometry of one or more components of the imagingsystem, information about a geometry of the image acquisition,information about one or more image acquisition parameters, or acombination thereof; c. generating, by the at least one computerprocessor, a 3D representation of a surface, a volume or combinationthereof, wherein the 3D representation of the surface, the volume orcombination thereof is at least in part derived from the informationabout the geometry of the one or more components of the imaging system,information about the geometry of the image acquisition, informationabout the one or more image acquisition parameters, or combinationthereof; d. generating, by the at least one computer processor, anaugmented view, the augmented view comprising the 3D representation ofthe surface, volume or combination thereof; and e. displaying, by anaugmented reality display device, the augmented view, wherein positionand orientation of the augmented view is defined relative to the one ormore components of the imaging system and is updated based on real timetracking information of the one or more components of the imagingsystem.
 23. The method of claim 22, wherein the imaging system isconfigured to acquire 2D, 3D, or 2D and 3D imaging data of the patient,and wherein the 2D, 3D, or 2D and 3D imaging data of the patient areacquired within the 3D representation of the surface, volume orcombination thereof.
 24. The method of claim 22, wherein the step ofgenerating the augmented view is before the step of acquiring 2D, 3D, or2D and 3D imaging data of the patient, or wherein the step of displayingthe augmented view is before the step of acquiring 2D, 3D, or 2D and 3Dimaging data of the patient.
 25. The method of claim 22, wherein theaugmented reality display device is a head mounted display, and whereinthe augmented view comprises a 3D stereoscopic view.